@community_library
Source: github.com/f/prompts.chat · Contributor: f
---
name: reading-plan-and-page-pace-planner
description: Plans your reading day by day so you finish every book before its book club, course or exam date - from page or word counts (or audiobook length), your reading speed, the minutes you have each weekday, trips and busy days. Spreads pages evenly, keeps buffer days before each deadline and weekly catch-up days, stops at chapter ends, warns when a deadline is out of reach and re-plans from where you are. Includes a stdlib Python planner.
---
# Reading Plan and Page Pace Planner
You help readers stop cramming 200 pages the night before book club. People know roughly how much time they have (half an hour on weekdays, more at the weekend) but not what that means in pages for a 245-page novel due in two weeks, a dense work book and an 880-page classic in the same month. You turn books, reading speed and free minutes into a concrete daily plan ("Tue: p. 1-27"), keep slack for real life, and say early and plainly when a deadline cannot be met and what would fix it.
## Files in this skill
- `scripts/reading_plan.py` - reads a plan JSON (books with pages, words or audiobook minutes, start page, difficulty, deadline, chapter starts; reading speed in words per minute or pages per hour; minutes per weekday; trips and exceptions; catch-up weekdays; buffer days); schedules earliest deadline first, spreads the load evenly (or as soon as possible), snaps daily stops to chapter ends, and prints a book summary, findings RP01-RP06, a day-by-day table with page ranges, weekly checkpoints and catch-up advice; `--from` + `--at "Title=page"` re-plans mid-way (Python 3.8+ standard library only)
- `references/reading-speed-and-estimates.md` - measuring your speed, words per page by book type, difficulty factors, audiobooks
- `references/pacing-and-catch-up.md` - buffers, catch-up days, what to do when you fall behind or a deadline is impossible
- `templates/reading-plan.md` - the plan to hand back (calendar-friendly)
- `examples/book-club.json` - November-December: a novel for a Thursday book club, a work book already started at p. 23, Middlemarch for a classics club, an audiobook with no deadline, Sunday catch-up days, a weekend trip and two unusual days
- `examples/book-club-plan.md` - the real planner output on that file
## Workflow
### 1. Collect the inputs
Ask for: each book with its page count (or word count; the edition matters), where they are now, the deadline and what it is (meeting, exam), whether it is easy or dense; their reading speed (or time a 10-minute test with `references/reading-speed-and-estimates.md`); realistic minutes per weekday, not ideal ones; known trips, late shifts and free days; which weekday should stay free as a catch-up day; and how many days before each meeting they want to be done (buffer, default 1-2).
### 2. Run the planner
```
python3 scripts/reading_plan.py examples/book-club.json
python3 scripts/reading_plan.py examples/book-club.json --mode asap
python3 scripts/reading_plan.py examples/book-club.json --from 2026-11-20 --at "Thinking in Systems=110"
python3 scripts/reading_plan.py plan.json --json
python3 scripts/reading_plan.py --self-test
```
Exit 0 = every deadline met, 1 = at least one book cannot be finished in time, 2 = bad input. Without a shell: minutes needed = pages x words per page / words per minute x difficulty; compare with the minutes available up to each finish-by date (deadline minus buffer, catch-up days excluded), earliest deadline first.
### 3. Read the result
- RP01 (high): a book does not fit before its deadline. The message gives the shortfall in pages and the extra minutes per reading day that would close it. When the plan is infeasible the planner reads at full pace before it, so earlier books finish early to free time.
- RP02 / RP04 (medium): the plan only works by eating the buffer, or uses over 90% of your time in its busiest stretch - one bad week breaks it.
- RP03 (low): days longer than the max session - split them. RP05/RP06 (info): finished books, close deadlines, audiobooks.
- Check the day table for heavy weekend days and the weekly checkpoints for "where should I be by Sunday".
### 4. Fix an impossible plan
Offer options in this order: start the long book now in parallel (move its deadline logic earlier), add 10-20 minutes on some days, use the catch-up days, switch part of it to the audiobook for commutes, skim or skip clearly marked sections, or accept arriving at the meeting part-way and say so. Re-run after each change. See `references/pacing-and-catch-up.md`.
### 5. Deliver
Fill `templates/reading-plan.md`: per-book summary, the week-by-week plan (dates and page ranges, ready for a calendar or a bookmark note), the catch-up rule, and the re-plan command. Offer a quick re-plan whenever they report their current page.
## Example result
On `examples/book-club.json` (240 wpm, 20-60 min a day, Sundays free for catch-up, 2 buffer days) *The Remains of the Day* is done on 12 Nov for the 19 Nov book club, stopping at chapter ends, and *Thinking in Systems* (dense, factor 1.3, from p. 23) on 27 Nov for the 1 Dec group. *Middlemarch* (316,000 words, about 22 hours) cannot be finished by the 17 Dec classics club: about 452 pages short, projected finish 11 Jan (RP01, high). Fix: about +17 minutes on each of the 42 reading days, or start it alongside the other two now. The 120-minute quiet Saturday is flagged as two sessions (RP03), and the Piranesi audiobook (6 h 40 at 1.25x) follows after Middlemarch.
## Rules
- Plan with the time people really have, and keep at least one buffer day before every meeting.
- Never quietly overfill days: if it does not fit, say which deadline fails, by how many pages, and what would fix it.
- Use the edition's page count; when only word counts are known, say that pages are estimates.
- Leave catch-up days empty in the plan - they are for falling behind, not for more reading.
- Reading should stay enjoyable: suggest skipping or switching formats before suggesting 2-hour sessions every day.
- Re-plan from the current page whenever the reader is behind or ahead; do not make them "catch up" to an old schedule.
FILE:references/reading-speed-and-estimates.md
# Reading speed and estimates
## Measure your speed (10 minutes)
1. Pick the actual book (or one like it), start at a chapter start, set a 10-minute timer and read normally.
2. Count the words on three full lines, divide by 3 = words per line; count the lines on a full page = lines per page.
3. words per page = words per line x lines per page; words per minute = pages read x words per page / 10.
Or simply time a few pages and use `"pages_per_hour"` instead of words per minute.
## Typical values
| Reader / material | Words per minute |
|---|---|
| Careful reading, dense non-fiction, second language | 150-200 |
| Average adult, fiction | 200-260 |
| Fast leisure reader | 300-400 |
Skimming is not reading: do not plan with speeds above 450.
| Book type | Words per page |
|---|---|
| Mass-market paperback | 300-350 |
| Trade paperback fiction | 250-300 |
| Hardcover non-fiction with notes | 350-450 |
| Large print, children's chapter books | 150-200 |
| E-book | use the word count (store page or tools that show it) and the default 275 words per page |
## Difficulty factor
`"difficulty"` multiplies reading time: 1.0 easy fiction, 1.2-1.4 dense non-fiction, classics with long sentences, technical books with diagrams, 1.5+ textbooks you take notes on, poetry, second-language reading.
## Page count vs word count
- Use the page count of the edition the reader holds; editions of the same book can differ by 30%.
- With both `"pages"` and `"words"`, the planner derives the words per page of that edition (Middlemarch 316,000 / 880 = 359).
- With only words, pages are estimates (words / words per page); say so in the plan.
## Audiobooks
- `"audio_minutes"` is the runtime, `"speed"` the playback speed: time needed = runtime / speed (400 min at 1.25x = 320 min).
- Audio fits commutes, chores and walks: say whether those minutes are extra or already counted in the daily minutes.
- Positions are shown as h:mm of the audio.
FILE:references/pacing-and-catch-up.md
# Pacing and catch-up
## How the planner schedules
- Earliest deadline first: the book with the nearest finish-by date (deadline minus buffer days) is read first; books without a deadline come last.
- Even mode (default): for each stretch up to the tightest deadline it uses the smallest equal share of your daily minutes that still finishes in time, so 30-minute days might become 20-minute days when there is room. After that deadline the next stretch is computed again.
- ASAP mode: uses all the minutes every day; books finish as early as possible.
- If a deadline cannot be met even at full pace, every day before it is planned at full pace and the book is flagged RP01.
- Daily stops move to a chapter end when one is within `chapter_snap_pages` (default 8) and a quarter of the day's pages.
## Buffers and catch-up days
- Buffer days (`buffer_days`, default 1) sit between the planned finish and the meeting: time to reread notes or absorb a bad week.
- Catch-up weekdays (`catch_up_days`, e.g. Sunday) get no planned reading. Use them only when behind; if you are on track they are free time.
- Exceptions (trips, late shifts, a quiet Saturday) override the weekday minutes for those dates.
## When you fall behind
1. Missed one day: read that day's pages on the next catch-up day, or add a third of them to each of the next three days.
2. Behind by more than two days: re-plan from today with `--from <today> --at "<Title>=<current page>"` instead of chasing the old plan.
3. Still not in time (RP01): pick one - start the long book in parallel now, add 10-20 minutes on a few days, use the audiobook for commutes, skim marked sections (prefaces, appendices, notes), or decide to arrive part-way and tell the group.
## Signs the plan is too tight
- RP04: over 90% of your time used in a stretch; any illness or late week breaks it.
- RP02: the book only fits by using the buffer days.
- Many days marked "2 sessions" (RP03): consider spreading reading across morning and evening on purpose.
## Book club tips
- Note the page or chapter the group will discuss if it is not the whole book, and use that as the book's `pages`.
- For long classics, ask the club whether the meeting covers the first half; plan two deadlines.
- Keep a sticky note in the book with the weekly checkpoints from the plan.
FILE:templates/reading-plan.md
# Reading plan: <name> (<start> to <last deadline>)
**Pace:** <wpm> words/min, about <pages> pages in a 30-minute session
**Time:** <minutes per weekday>; catch-up day(s): <day>; buffer: <n> day(s) before each meeting
**Verdict:** <every deadline is met | BOOK cannot be finished by DATE - see options>
## Books
| Book | Pages left | Time needed | Meeting / deadline | Finish by | Planned finish |
|---|---|---|---|---|---|
| <title> | <pp> | <h:mm> | <date - event> | <date> | <date> |
## Week by week
### Week of <Mon date>
- Mon: <Book p. a-b>
- Tue: <...>
- Sun: catch-up / free
**Checkpoint Sunday night:** <Book p. X>
## If a deadline does not fit
- Shortfall: <pages> pages of <book> by <date>
- Options: <+N min on reading days | start now in parallel | audiobook for commutes | skim chapters x-y | arrive part-way>
- Chosen: <...>
## Falling behind
- Missed a day? Use <catch-up day> or add <n> pages to the next three days.
- More than two days behind: re-plan from today (`--from <date> --at "<title>=<page>"`).
FILE:examples/book-club.json
{
"name": "Reading plan: Nov-Dec 2026 (two book clubs and a work book)",
"start": "2026-11-03",
"reader": {"words_per_minute": 240, "words_per_page": 280},
"minutes": {"mon": 30, "tue": 30, "wed": 20, "thu": 30, "fri": 25, "sat": 60, "sun": 45},
"catch_up_days": ["sun"],
"buffer_days": 2,
"max_session_minutes": 60,
"chapter_snap_pages": 8,
"exceptions": [
{"dates": "2026-11-13..2026-11-15", "minutes": 0, "note": "weekend trip"},
{"dates": "2026-11-26", "minutes": 15, "note": "late shift"},
{"dates": "2026-12-05", "minutes": 120, "note": "quiet Saturday"}
],
"books": [
{"title": "The Remains of the Day", "pages": 245, "words_per_page": 265, "deadline": "2026-11-19", "event": "Thursday book club",
"chapters": [1, 9, 45, 101, 125, 165, 212, 239]},
{"title": "Thinking in Systems", "pages": 218, "start_page": 23, "difficulty": 1.3, "deadline": "2026-12-01", "event": "team reading group"},
{"title": "Middlemarch", "words": 316000, "pages": 880, "deadline": "2026-12-17", "event": "classics club"},
{"title": "Piranesi (audiobook)", "audio_minutes": 400, "speed": 1.25}
]
}
FILE:examples/book-club-plan.md
# Reading plan: Nov-Dec 2026 (two book clubs and a work book)
Start 2026-11-03, even mode, 2 buffer day(s) before each deadline, catch-up days: sun. Planned reading: 36 h 44 min over 81 days.
**Verdict: 1 book(s) cannot be finished by their deadline - change the plan**
## Books
| Book | Left | Est. time | Deadline | Finish by | Planned finish | Days spare | Avg per reading day |
|---|---|---|---|---|---|---|---|
| The Remains of the Day | 245 pp | 4:31 h | 2026-11-19 (Thursday book club) | 2026-11-17 | 2026-11-12 | 7 | 27 pp |
| Thinking in Systems | 196 pp | 4:57 h | 2026-12-01 (team reading group) | 2026-11-29 | 2026-11-27 | 4 | 16 pp |
| Middlemarch | 880 pp | 21:57 h | 2026-12-17 (classics club) | 2026-12-15 | 2027-01-11 | -25 | 23 pp |
| Piranesi (audiobook) | 6:40 audio | 5:20 h | - | - | 2027-01-22 | - | 0:36 |
## Findings
- **HIGH RP01** Middlemarch cannot be finished by 2026-12-17 (classics club): about 452 pages short at your pace; projected finish 2027-01-11. Options: +17 min on each of the 42 reading days until then, read faster, skim or skip about 452 pages, or start it earlier
- **LOW RP03** 1 day(s) plan more than 60 minutes (e.g. 2026-12-05: 120 min) - split into two sessions
- **INFO RP06** Piranesi (audiobook) is an audiobook: the plan counts listening time at 1.25x speed; commutes and chores can cover it
## Day by day
| Date | Day | Minutes | Read | Note |
|---|---|---|---|---|
| 2026-11-03 | Tue | 30 | The Remains of the Day p. 1-27 (27 pp) | |
| 2026-11-04 | Wed | 19 | The Remains of the Day p. 28-44 (17 pp) | |
| 2026-11-05 | Thu | 30 | The Remains of the Day p. 45-71 (27 pp) | |
| 2026-11-06 | Fri | 25 | The Remains of the Day p. 72-94 (23 pp) | |
| 2026-11-07 | Sat | 60 | The Remains of the Day p. 95-148 (54 pp) | |
| 2026-11-08 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-11-09 | Mon | 30 | The Remains of the Day p. 149-175 (27 pp) | |
| 2026-11-10 | Tue | 30 | The Remains of the Day p. 176-202 (27 pp) | |
| 2026-11-11 | Wed | 20 | The Remains of the Day p. 203-221 (19 pp) | |
| 2026-11-12 | Thu | 30 | The Remains of the Day p. 222-245 (24 pp); Thinking in Systems p. 23-24 (2 pp) | |
| 2026-11-13 | Fri | - | - | weekend trip |
| 2026-11-14 | Sat | - | - | weekend trip |
| 2026-11-15 | Sun | - | - | weekend trip |
| 2026-11-16 | Mon | 30 | Thinking in Systems p. 25-44 (20 pp) | |
| 2026-11-17 | Tue | 30 | Thinking in Systems p. 45-64 (20 pp) | finish-by The Remains of the Day |
| 2026-11-18 | Wed | 20 | Thinking in Systems p. 65-77 (13 pp) | |
| 2026-11-19 | Thu | 30 | Thinking in Systems p. 78-97 (20 pp) | DEADLINE The Remains of the Day - Thursday book club |
| 2026-11-20 | Fri | 25 | Thinking in Systems p. 98-113 (16 pp) | |
| 2026-11-21 | Sat | 60 | Thinking in Systems p. 114-153 (40 pp) | |
| 2026-11-22 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-11-23 | Mon | 30 | Thinking in Systems p. 154-172 (19 pp) | |
| 2026-11-24 | Tue | 30 | Thinking in Systems p. 173-192 (20 pp) | |
| 2026-11-25 | Wed | 20 | Thinking in Systems p. 193-205 (13 pp) | |
| 2026-11-26 | Thu | 15 | Thinking in Systems p. 206-215 (10 pp) | late shift |
| 2026-11-27 | Fri | 25 | Thinking in Systems p. 216-218 (3 pp); Middlemarch p. 1-14 (14 pp) | |
| 2026-11-28 | Sat | 60 | Middlemarch p. 15-54 (40 pp) | |
| 2026-11-29 | Sun | - | - | catch-up / spare (45 min available); finish-by Thinking in Systems |
| 2026-11-30 | Mon | 30 | Middlemarch p. 55-74 (20 pp) | |
| 2026-12-01 | Tue | 30 | Middlemarch p. 75-94 (20 pp) | DEADLINE Thinking in Systems - team reading group |
| 2026-12-02 | Wed | 20 | Middlemarch p. 95-107 (13 pp) | |
| 2026-12-03 | Thu | 30 | Middlemarch p. 108-128 (21 pp) | |
| 2026-12-04 | Fri | 25 | Middlemarch p. 129-144 (16 pp) | |
| 2026-12-05 | Sat | 120 | Middlemarch p. 145-224 (80 pp) | quiet Saturday; 2 sessions |
| 2026-12-06 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-12-07 | Mon | 30 | Middlemarch p. 225-244 (20 pp) | |
| 2026-12-08 | Tue | 30 | Middlemarch p. 245-265 (21 pp) | |
| 2026-12-09 | Wed | 20 | Middlemarch p. 266-278 (13 pp) | |
| 2026-12-10 | Thu | 30 | Middlemarch p. 279-298 (20 pp) | |
| 2026-12-11 | Fri | 25 | Middlemarch p. 299-315 (17 pp) | |
| 2026-12-12 | Sat | 60 | Middlemarch p. 316-355 (40 pp) | |
| 2026-12-13 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-12-14 | Mon | 30 | Middlemarch p. 356-375 (20 pp) | |
| 2026-12-15 | Tue | 30 | Middlemarch p. 376-395 (20 pp) | finish-by Middlemarch |
| 2026-12-16 | Wed | 20 | Middlemarch p. 396-408 (13 pp) | |
| 2026-12-17 | Thu | 30 | Middlemarch p. 409-428 (20 pp) | DEADLINE Middlemarch - classics club |
| 2026-12-18 | Fri | 25 | Middlemarch p. 429-445 (17 pp) | |
| 2026-12-19 | Sat | 60 | Middlemarch p. 446-485 (40 pp) | |
| 2026-12-20 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-12-21 | Mon | 30 | Middlemarch p. 486-505 (20 pp) | |
| 2026-12-22 | Tue | 30 | Middlemarch p. 506-525 (20 pp) | |
| 2026-12-23 | Wed | 20 | Middlemarch p. 526-539 (14 pp) | |
| 2026-12-24 | Thu | 30 | Middlemarch p. 540-559 (20 pp) | |
| 2026-12-25 | Fri | 25 | Middlemarch p. 560-575 (16 pp) | |
| 2026-12-26 | Sat | 60 | Middlemarch p. 576-615 (40 pp) | |
| 2026-12-27 | Sun | - | - | catch-up / spare (45 min available) |
| 2026-12-28 | Mon | 30 | Middlemarch p. 616-635 (20 pp) | |
| 2026-12-29 | Tue | 30 | Middlemarch p. 636-656 (21 pp) | |
| 2026-12-30 | Wed | 20 | Middlemarch p. 657-669 (13 pp) | |
| 2026-12-31 | Thu | 30 | Middlemarch p. 670-689 (20 pp) | |
| 2027-01-01 | Fri | 25 | Middlemarch p. 690-706 (17 pp) | |
| 2027-01-02 | Sat | 60 | Middlemarch p. 707-746 (40 pp) | |
| 2027-01-03 | Sun | - | - | catch-up / spare (45 min available) |
| 2027-01-04 | Mon | 30 | Middlemarch p. 747-766 (20 pp) | |
| 2027-01-05 | Tue | 30 | Middlemarch p. 767-786 (20 pp) | |
| 2027-01-06 | Wed | 20 | Middlemarch p. 787-799 (13 pp) | |
| 2027-01-07 | Thu | 30 | Middlemarch p. 800-819 (20 pp) | |
| 2027-01-08 | Fri | 25 | Middlemarch p. 820-836 (17 pp) | |
| 2027-01-09 | Sat | 60 | Middlemarch p. 837-876 (40 pp) | |
| 2027-01-10 | Sun | - | - | catch-up / spare (45 min available) |
| 2027-01-11 | Mon | 30 | Middlemarch p. 877-880 (4 pp); Piranesi (audiobook) 0:00-0:30 | |
| 2027-01-12 | Tue | 30 | Piranesi (audiobook) 0:30-1:08 | |
| 2027-01-13 | Wed | 20 | Piranesi (audiobook) 1:08-1:33 | |
| 2027-01-14 | Thu | 30 | Piranesi (audiobook) 1:33-2:10 | |
| 2027-01-15 | Fri | 25 | Piranesi (audiobook) 2:10-2:41 | |
| 2027-01-16 | Sat | 60 | Piranesi (audiobook) 2:41-3:56 | |
| 2027-01-17 | Sun | - | - | catch-up / spare (45 min available) |
| 2027-01-18 | Mon | 30 | Piranesi (audiobook) 3:56-4:34 | |
| 2027-01-19 | Tue | 30 | Piranesi (audiobook) 4:34-5:11 | |
| 2027-01-20 | Wed | 20 | Piranesi (audiobook) 5:11-5:36 | |
| 2027-01-21 | Thu | 30 | Piranesi (audiobook) 5:36-6:14 | |
| 2027-01-22 | Fri | 21 | Piranesi (audiobook) 6:14-6:40 | |
## Weekly checkpoints (where you should be by Sunday night)
- 2026-11-08: The Remains of the Day p. 148
- 2026-11-15: Thinking in Systems p. 24
- 2026-11-22: Thinking in Systems p. 153
- 2026-11-29: Middlemarch p. 54
- 2026-12-06: Middlemarch p. 224
- 2026-12-13: Middlemarch p. 355
- 2026-12-20: Middlemarch p. 485
- 2026-12-27: Middlemarch p. 615
- 2027-01-03: Middlemarch p. 746
- 2027-01-10: Middlemarch p. 876
- 2027-01-17: Piranesi (audiobook) 3:56
## If you fall behind
- The Remains of the Day: a missed day costs about 27 pages. Make them up on the spare day(s) 2026-11-08, or add 10 pages to each of the next 3 reading days. Re-plan any time with `--from <date> --at "The Remains of the Day=<page>"`.
- Thinking in Systems: a missed day costs about 16 pages. Make them up on the spare day(s) 2026-11-22, 2026-11-29, or add 6 pages to each of the next 3 reading days. Re-plan any time with `--from <date> --at "Thinking in Systems=<page>"`.
- Middlemarch: a missed day costs about 23 pages. Make them up on the spare day(s) 2026-11-29, 2026-12-06, 2026-12-13, or add 8 pages to each of the next 3 reading days. Re-plan any time with `--from <date> --at "Middlemarch=<page>"`.
FILE:scripts/reading_plan.py
#!/usr/bin/env python3
"""Reading plan and page pace planner (stdlib only, Python 3.8+).
Turns a list of books (pages, word counts or audiobook length), your reading speed and the
minutes you have each day into a day-by-day plan with page ranges that meets every deadline
(book club, course, exam), keeps buffer days before each deadline and catch-up days in the
week, spreads the load evenly and tells you what to do if you fall behind.
Usage:
reading_plan.py PLAN.json [--mode even|asap] [--from YYYY-MM-DD] [--at "Title=page"]... [--json]
reading_plan.py --self-test
Exit: 0 = every deadline is met, 1 = at least one book cannot be finished by its deadline,
2 = bad input.
"""
import argparse, datetime, json, math, re, sys
DAYS = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]
SEV = {"high": 0, "medium": 1, "low": 2, "info": 3}
class InputError(Exception):
pass
def pdate(s, what):
try:
return datetime.date.fromisoformat(str(s))
except ValueError:
raise InputError("%s must be YYYY-MM-DD, got %r" % (what, s))
def expand_dates(spec):
spec = str(spec)
if ".." in spec:
a, b = spec.split("..", 1)
a, b = pdate(a.strip(), "date range"), pdate(b.strip(), "date range")
if b < a:
raise InputError("date range %s ends before it starts" % spec)
return [a + datetime.timedelta(days=i) for i in range((b - a).days + 1)]
return [pdate(spec.strip(), "date")]
# ---------------------------------------------------------------- model
def load(cfg, start_override=None, at=None):
if not isinstance(cfg, dict) or not isinstance(cfg.get("books"), list) or not cfg["books"]:
raise InputError("plan needs a non-empty 'books' list")
reader = cfg.get("reader") or {}
wpm = float(reader.get("words_per_minute", 230))
wpp_default = float(reader.get("words_per_page", 275))
pph = reader.get("pages_per_hour")
if wpm <= 0 or wpp_default <= 0:
raise InputError("words_per_minute and words_per_page must be positive")
start = pdate(start_override or cfg.get("start") or datetime.date.today().isoformat(), "start")
mins = cfg.get("minutes")
if isinstance(mins, (int, float)):
mins = {d: mins for d in DAYS}
if not isinstance(mins, dict):
raise InputError("'minutes' must be a number or {mon: .., tue: ..}")
mins = {k.lower()[:3]: float(v) for k, v in mins.items()}
bad = set(mins) - set(DAYS)
if bad:
raise InputError("unknown weekday key(s) in minutes: %s" % ", ".join(sorted(bad)))
catch = {d.lower()[:3] for d in cfg.get("catch_up_days", [])}
exc = {}
for e in cfg.get("exceptions", []):
for d in expand_dates(e.get("dates") or e.get("date")):
exc[d] = (float(e.get("minutes", 0)), e.get("note", ""))
books = []
at = at or {}
for i, b in enumerate(cfg["books"], 1):
t = b.get("title") or "Book %d" % i
if "audio_minutes" in b:
speed = float(b.get("speed", 1.0))
total = float(b["audio_minutes"])
unit_min = 1.0 / speed # minutes of your time per minute of audio
units, kind = total, "audio"
done = float(at.get(t, b.get("start_minute", 0)))
else:
wpp = float(b.get("words_per_page") or (float(b["words"]) / float(b["pages"]) if b.get("words") and b.get("pages") else wpp_default))
pages = float(b["pages"]) if b.get("pages") else (float(b["words"]) / wpp if b.get("words") else None)
if not pages or pages <= 0:
raise InputError("book %r needs pages, words or audio_minutes" % t)
diff = float(b.get("difficulty", 1.0))
if pph and not b.get("words"):
unit_min = 60.0 / float(pph) * diff
else:
unit_min = wpp / wpm * diff
units, kind = round(pages), "pages"
sp = at.get(t, b.get("start_page", 1))
done = max(0.0, float(sp) - 1)
dl = pdate(b["deadline"], "deadline of %r" % t) if b.get("deadline") else None
chapters = sorted(int(c) for c in b.get("chapters", []))
books.append({"title": t, "units": float(units), "kind": kind, "unit_min": unit_min, "done": min(done, float(units)),
"deadline": dl, "event": b.get("event", ""), "chapters": chapters, "order": i})
buf = int(cfg.get("buffer_days", 1))
return {"start": start, "mins": mins, "catch": catch, "exc": exc, "books": books, "buffer": buf,
"max_session": float(cfg.get("max_session_minutes", 0) or 0), "snap": int(cfg.get("chapter_snap_pages", 8)),
"name": cfg.get("name", "Reading plan"), "wpm": wpm}
def capacity(m, d):
"""(planned minutes, available minutes incl. catch-up, note)."""
if d in m["exc"]:
v, note = m["exc"][d]
return v, v, note or ("no reading" if v == 0 else "")
wd = DAYS[d.weekday()]
v = m["mins"].get(wd, 0.0)
if wd in m["catch"]:
return 0.0, v, "catch-up day"
return v, v, ""
def plan(m, mode="even"):
F = []
books = [b for b in m["books"]]
for b in books:
b["need"] = (b["units"] - b["done"]) * b["unit_min"]
if b["need"] <= 0:
F.append(("info", "RP05", "%s is already finished - left out" % b["title"]))
b["target"] = None
if b["deadline"] and b["deadline"] < m["start"]:
if b["need"] > 0:
F.append(("medium", "RP05", "%s: deadline %s has already passed - planned without a deadline" % (b["title"], b["deadline"])))
b["deadline"] = None
if b["deadline"]:
b["target"] = b["deadline"] - datetime.timedelta(days=m["buffer"])
if b["target"] < m["start"]:
b["target"] = m["start"]
F.append(("medium", "RP05", "%s: deadline %s is too close for %d buffer day(s) - planning to the deadline" % (b["title"], b["deadline"], m["buffer"])))
todo = [b for b in books if b["need"] > 0]
dated = sorted([b for b in todo if b["target"]], key=lambda b: (b["target"], b["order"]))
undated = [b for b in todo if not b["target"]]
seq = dated + undated
last = max([b["deadline"] for b in dated] or [m["start"]])
# horizon: enough days for everything at full capacity, plus slack
weekly = sum(m["mins"].get(d, 0) for d in DAYS if d not in m["catch"]) or 1
total_need = sum(b["need"] for b in seq)
horizon_end = max(last, m["start"]) + datetime.timedelta(days=int(total_need / weekly * 7) + 60)
days, d = [], m["start"]
while d <= horizon_end:
p, a, note = capacity(m, d)
days.append({"date": d, "cap": p, "avail": a, "note": note, "planned": 0.0, "items": []})
d += datetime.timedelta(days=1)
idx = {x["date"]: i for i, x in enumerate(days)}
# load factor per segment (even mode): smallest uniform share of capacity that meets every target
load = [1.0] * len(days)
if mode == "even" and dated:
seg_start, k0 = 0, 0
while k0 < len(dated):
best, bk = 0.0, k0
cum = 0.0
for k in range(k0, len(dated)):
cum += dated[k]["need"]
ti = idx[dated[k]["target"]]
cap = sum(x["cap"] for x in days[seg_start:ti + 1])
r = cum / cap if cap > 0 else float("inf")
if r >= best:
best, bk = r, k
L = min(best * 1.03, 1.0) # 3% headroom for chapter-end stops and rounding
end = idx[dated[bk]["target"]]
for i in range(seg_start, end + 1):
load[i] = L
seg_start, k0 = end + 1, bk + 1
# after the last deadline: undated books at the average load of the plan so far (at least 0.5)
tail = max(0.5, max(load[:seg_start] or [1.0]))
for i in range(seg_start, len(days)):
load[i] = tail if undated else 1.0
# allocate (earliest deadline first), tracking each book's position
qi = 0
pos = {b["title"]: b["done"] for b in seq}
for i, day in enumerate(days):
budget = day["cap"] * load[i]
while budget > 1e-9 and qi < len(seq):
b = seq[qi]
remaining = (b["units"] - pos[b["title"]]) * b["unit_min"]
use = min(budget, remaining)
before = pos[b["title"]]
after = before + use / b["unit_min"]
if after > b["units"] - 1e-6:
after = b["units"]
elif b["kind"] == "pages":
snapped = snap(after, b, m["snap"], use / b["unit_min"])
if snapped != after:
after = snapped
use = (after - before) * b["unit_min"]
budget = use # stop at the chapter end: the day ends here
pos[b["title"]] = after
day["items"].append({"title": b["title"], "from": before, "to": after, "kind": b["kind"], "minutes": use})
day["planned"] += use
budget -= use
if after >= b["units"] - 1e-6:
b["finish"] = day["date"]; qi += 1
elif budget <= 1e-9 or use <= 1e-9:
break
if qi >= len(seq):
break
days = [x for x in days if x["date"] <= max([b.get("finish", m["start"]) for b in seq] + [last])]
for b in seq:
b.setdefault("finish", None)
# findings
for b in dated:
f = b["finish"]
if f is None or f > b["deadline"]:
avail = [x for x in days if x["date"] <= b["deadline"]]
share = sum(it["minutes"] for x in avail for it in x["items"] if it["title"] == b["title"])
short_min = b["need"] - share
# also count catch-up/buffer time that would be available up to the deadline
reading_days = sum(1 for x in avail if x["avail"] > 0) or 1
short_units = short_min / b["unit_min"]
msg = "%s cannot be finished by %s%s: about %d %s short at your pace; projected finish %s. Options: +%d min on each of the %d reading days until then, read faster, skim or skip about %d %s, or start it earlier" % (
b["title"], b["deadline"], " (%s)" % b["event"] if b["event"] else "", math.ceil(short_units), "pages" if b["kind"] == "pages" else "audio minutes",
f or "beyond the plan", math.ceil(short_min / reading_days), reading_days, math.ceil(short_units), "pages" if b["kind"] == "pages" else "audio minutes")
F.append(("high", "RP01", msg))
elif f > b["target"]:
F.append(("medium", "RP02", "%s finishes %s, inside the %d buffer day(s) before %s - no room for a missed day" % (b["title"], f, m["buffer"], b["deadline"])))
if mode == "even":
peak = max(load) if load else 0
if 0.9 <= peak <= 1.0 and not any(f[1] == "RP01" for f in F):
F.append(("medium", "RP04", "the plan uses %d%% of your planned reading time in its busiest stretch - one bad week will push you into the buffer" % round(peak * 100)))
if m["max_session"]:
long = [x for x in days if x["planned"] > m["max_session"] + 0.5]
if long:
F.append(("low", "RP03", "%d day(s) plan more than %d minutes (e.g. %s: %d min) - split into two sessions" % (
len(long), m["max_session"], long[0]["date"], round(long[0]["planned"]))))
for b in seq:
if b["kind"] == "audio":
F.append(("info", "RP06", "%s is an audiobook: the plan counts listening time at %sx speed; commutes and chores can cover it" % (b["title"], ("%.2f" % (1 / b["unit_min"])).rstrip("0").rstrip("."))))
F.sort(key=lambda f: (SEV[f[0]], f[1]))
return {"days": days, "books": seq, "all_books": books, "findings": F, "load": load}
def snap(after, b, snap_pages, day_units):
"""Move a day's stopping point to a chapter end if one is close (within snap_pages and 25% of the day)."""
if not b["chapters"] or snap_pages <= 0:
return after
lim = min(snap_pages, max(1.0, day_units * 0.25))
ends = [c - 1 for c in b["chapters"] if c - 1 > 0]
best = None
for e in ends:
if abs(e - after) <= lim and e > after - day_units + 0.5 and (best is None or abs(e - after) < abs(best - after)):
best = e
return float(best) if best is not None else after
# ---------------------------------------------------------------- rendering
def fmt_item(it):
if it["kind"] == "audio":
return "%s %s-%s" % (it["title"], hm(it["from"]), hm(it["to"]))
a = int(round(it["from"])) + 1
z = int(round(it["to"]))
if z < a:
z = a
return "%s p. %d-%d (%d pp)" % (it["title"], a, z, z - a + 1)
def hm(x):
x = int(round(x))
return "%d:%02d" % (x // 60, x % 60)
def checkpoints(r):
out = []
pos = {}
for x in r["days"]:
for it in x["items"]:
pos[it["title"]] = it
if x["date"].weekday() == 6 and pos: # Sundays
out.append((x["date"], "; ".join("%s %s" % (t, ("p. %d" % round(it["to"]) if it["kind"] == "pages" else hm(it["to"]))) for t, it in pos.items() if it["to"] < next(b["units"] for b in r["books"] if b["title"] == t) - 1e-6) or "all current books done"))
return out
def report(r, m, mode):
hi = [f for f in r["findings"] if f[0] == "high"]
out = ["# %s" % m["name"], ""]
tot_min = sum(x["planned"] for x in r["days"])
out.append("Start %s, %s mode, %d buffer day(s) before each deadline, catch-up days: %s. Planned reading: %d h %02d min over %d days." % (
m["start"], mode, m["buffer"], ", ".join(sorted(m["catch"], key=DAYS.index)) or "none", tot_min // 60, tot_min % 60, len(r["days"])))
out += ["", "**Verdict: %s**" % ("%d book(s) cannot be finished by their deadline - change the plan" % len(hi) if hi else "every deadline is met"), ""]
out += ["## Books", "", "| Book | Left | Est. time | Deadline | Finish by | Planned finish | Days spare | Avg per reading day |", "|---|---|---|---|---|---|---|---|"]
for b in r["all_books"]:
left = b["units"] - b["done"]
rd = [x for x in r["days"] if any(it["title"] == b["title"] for it in x["items"])]
avg = left / len(rd) if rd else 0
spare = (b["deadline"] - b["finish"]).days if b.get("finish") and b["deadline"] else None
out.append("| %s | %s | %s | %s | %s | %s | %s | %s |" % (
b["title"], ("%d pp" % round(left)) if b["kind"] == "pages" else hm(left) + " audio", hm(b["need"]) + " h",
"%s %s" % (b["deadline"], "(%s)" % b["event"] if b["event"] else "") if b["deadline"] else "-",
b["target"] or "-", b.get("finish") or ("done" if b["need"] <= 0 else "NOT IN TIME"),
"-" if spare is None else spare, ("%.0f pp" % avg) if b["kind"] == "pages" else hm(avg)))
out += ["", "## Findings", ""]
out += ["- **%s %s** %s" % (s.upper(), c, msg) for s, c, msg in r["findings"]] or ["- none"]
out += ["", "## Day by day", "", "| Date | Day | Minutes | Read | Note |", "|---|---|---|---|---|"]
events = {}
for b in r["all_books"]:
if b["deadline"]:
events.setdefault(b["deadline"], []).append("DEADLINE %s%s" % (b["title"], " - " + b["event"] if b["event"] else ""))
if b["target"] and b["target"] != b["deadline"]:
events.setdefault(b["target"], []).append("finish-by %s" % b["title"])
for x in r["days"]:
note = [x["note"]] if x["note"] else []
if x["note"] == "catch-up day" or (x["planned"] < 0.5 and x["avail"] > 0):
note = ["catch-up / spare (%d min available)" % x["avail"]] if x["avail"] > 0 else note
note += events.get(x["date"], [])
if m["max_session"] and x["planned"] > m["max_session"] + 0.5:
note.append("2 sessions")
out.append("| %s | %s | %s | %s | %s |" % (x["date"], x["date"].strftime("%a"), round(x["planned"]) if x["planned"] >= 0.5 else "-",
"; ".join(fmt_item(it) for it in x["items"]) or "-", "; ".join(note)))
cps = checkpoints(r)
if cps:
out += ["", "## Weekly checkpoints (where you should be by Sunday night)", ""]
out += ["- %s: %s" % (d, s) for d, s in cps]
out += ["", "## If you fall behind", ""]
for b in r["books"]:
if b["kind"] != "pages" or not b["deadline"]:
continue
rd = [x for x in r["days"] if any(it["title"] == b["title"] for it in x["items"])]
if not rd:
continue
day_pp = sum(it["to"] - it["from"] for x in rd for it in x["items"] if it["title"] == b["title"]) / len(rd)
spare = [x for x in r["days"] if x["date"] <= b["deadline"] and x["date"] >= rd[0]["date"] and x["avail"] > 0 and x["planned"] < 0.5]
out.append("- %s: a missed day costs about %d pages. Make them up on %s, or add %d pages to each of the next 3 reading days. Re-plan any time with `--from <date> --at \"%s=<page>\"`." % (
b["title"], round(day_pp), ("the spare day(s) " + ", ".join(str(x["date"]) for x in spare[:3])) if spare else "no spare day before the deadline", math.ceil(day_pp / 3), b["title"]))
out.append("")
return "\n".join(out)
# ---------------------------------------------------------------- self-test
def self_test():
tests = []
def t(name, cond):
tests.append((name, bool(cond)))
def run(cfg, mode="even", **kw):
m = load(cfg, **kw)
return plan(m, mode), m
base = {"start": "2026-11-02", "reader": {"words_per_minute": 250, "words_per_page": 250}, "minutes": 30, "buffer_days": 0}
r, m = run(dict(base, books=[{"title": "A", "pages": 60, "deadline": "2026-11-08"}]), "asap")
t("asap: 30 pages/day at 1 page/min finishes day 2", r["books"][0]["finish"] == datetime.date(2026, 11, 3))
r, m = run(dict(base, books=[{"title": "A", "pages": 70, "deadline": "2026-11-08"}]))
t("even: spread over the week (~10 min/day + 3% headroom)", r["books"][0]["finish"] >= datetime.date(2026, 11, 7) and 10 <= r["days"][0]["planned"] <= 10.31)
r, m = run(dict(base, books=[{"title": "A", "pages": 300, "deadline": "2026-11-08"}]))
t("infeasible -> RP01 high", any(f[1] == "RP01" and f[0] == "high" for f in r["findings"]))
t("RP01 suggests extra minutes", any("+13 min" in f[2] for f in r["findings"] if f[1] == "RP01"))
r, m = run(dict(base, buffer_days=2, books=[{"title": "A", "pages": 70, "deadline": "2026-11-08"}]))
t("buffer days: finish 2 days early", r["books"][0]["finish"] == datetime.date(2026, 11, 6))
r, m = run(dict(base, catch_up_days=["sun"], books=[{"title": "A", "pages": 60, "deadline": "2026-11-08"}]))
t("catch-up Sunday gets no planned reading", r["days"][-1]["planned"] == 0 and r["books"][0]["finish"] == datetime.date(2026, 11, 7))
r, m = run(dict(base, exceptions=[{"dates": "2026-11-03..2026-11-04", "minutes": 0, "note": "trip"}], books=[{"title": "A", "pages": 50, "deadline": "2026-11-08"}]))
t("exception range has no reading", all(x["planned"] == 0 for x in r["days"][1:3]))
two = [{"title": "Late", "pages": 30, "deadline": "2026-11-20"}, {"title": "Soon", "pages": 60, "deadline": "2026-11-04"}]
r, m = run(dict(base, books=two))
t("earliest deadline first", r["days"][0]["items"][0]["title"] == "Soon")
t("tight segment then lighter load", r["load"][0] > 0.6 and r["load"][5] < 0.2)
r, m = run(dict(base, books=[{"title": "A", "pages": 70, "deadline": "2026-11-08"}]), at={"A": 36})
t("--at resumes from page", abs(sum(x["planned"] for x in r["days"]) - 35) < 0.01)
r, m = run(dict(base, books=[{"title": "W", "words": 25000, "deadline": "2026-11-08"}]))
t("word count -> pages via words_per_page", r["all_books"][0]["units"] == 100)
r, m = run(dict(base, books=[{"title": "H", "pages": 70, "difficulty": 1.5, "deadline": "2026-11-08"}]))
t("difficulty scales time", abs(r["books"][0]["need"] - 105) < 0.01)
r, m = run(dict(base, books=[{"title": "Au", "audio_minutes": 300, "speed": 1.5, "deadline": "2026-11-08"}]))
t("audiobook at 1.5x = 200 min", abs(r["books"][0]["need"] - 200) < 0.01 and any(f[1] == "RP06" for f in r["findings"]))
r, m = run(dict(base, books=[{"title": "C", "pages": 70, "deadline": "2026-11-08", "chapters": [1, 12, 23, 31, 42, 55]}]))
ends = [round(it["to"]) for x in r["days"] for it in x["items"]]
t("chapter snapping stops at chapter ends", 11 in ends and 22 in ends and ends[-1] == 70)
t("one item per book per day", all(len({it["title"] for it in x["items"]}) == len(x["items"]) for x in r["days"]))
t("page ranges do not overlap", all(fmt_item(it) for x in r["days"] for it in x["items"]) and
[int(round(it["from"])) for x in r["days"] for it in x["items"]][1:] == [int(round(it["to"])) for x in r["days"] for it in x["items"]][:-1])
r, m = run(dict(base, max_session_minutes=20, minutes=60, books=[{"title": "A", "pages": 300, "deadline": "2026-11-08"}]))
t("long sessions -> RP03", any(f[1] == "RP03" for f in r["findings"]))
r, m = run(dict(base, books=[{"title": "A", "pages": 200, "deadline": "2026-11-08"}]))
t("tight plan -> RP04 or RP01", any(f[1] in ("RP04", "RP01") for f in r["findings"]))
r, m = run(dict(base, books=[{"title": "A", "pages": 30}, {"title": "B", "pages": 60, "deadline": "2026-11-05"}]))
t("undated book after dated ones", r["books"][-1]["title"] == "A" and r["books"][-1]["finish"])
r, m = run(dict(base, books=[{"title": "A", "pages": 20, "start_page": 21}]))
t("finished book -> RP05", any(f[1] == "RP05" for f in r["findings"]))
t("pages_per_hour", abs(load(dict(base, reader={"pages_per_hour": 30}, books=[{"title": "P", "pages": 30}]))["books"][0]["unit_min"] - 2) < 1e-9)
for bad in [dict(base, books=[]), dict(base, books=[{"title": "x"}]), dict(base, minutes={"funday": 3}, books=[{"title": "a", "pages": 3}]),
dict(base, books=[{"title": "a", "pages": 3, "deadline": "2026-13-01"}])]:
try:
run(bad); okk = False
except InputError:
okk = True
t("bad input rejected: %s" % (list(bad.get("minutes", {})) if isinstance(bad.get("minutes"), dict) else (bad["books"][0] if bad["books"] else "no books")), okk)
r, m = run(dict(base, books=[{"title": "Old", "pages": 30, "deadline": "2026-10-01"}]))
t("passed deadline -> RP05, planned without deadline", any(f[1] == "RP05" for f in r["findings"]) and r["books"][0]["finish"])
rep = report(*run(dict(base, books=two)), "even")
t("report has verdict, day table and catch-up advice", "Verdict" in rep and "| Date |" in rep and "If you fall behind" in rep)
for name, okk in tests:
print(("PASS " if okk else "FAIL ") + name)
n = sum(1 for _, okk in tests if okk)
print("%d/%d passed" % (n, len(tests)))
return 0 if n == len(tests) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("plan", nargs="?")
ap.add_argument("--mode", choices=["even", "asap"], default="even")
ap.add_argument("--from", dest="start", help="re-plan from this date (YYYY-MM-DD)")
ap.add_argument("--at", action="append", default=[], help='"Title=page" (or "Title=minute" for audiobooks): where you are now')
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.plan:
ap.error("plan JSON required (or --self-test)")
try:
try:
cfg = json.load(open(a.plan, encoding="utf-8"))
except (OSError, ValueError) as e:
raise InputError(str(e))
at = {}
for s in a.at:
if "=" not in s:
raise InputError("--at needs Title=page, got %r" % s)
k, v = s.rsplit("=", 1)
try:
at[k.strip()] = float(v)
except ValueError:
raise InputError("--at page must be a number, got %r" % v)
titles = {b.get("title") for b in cfg.get("books", [])} if isinstance(cfg, dict) else set()
for k in at:
if k not in titles:
raise InputError("--at: no book titled %r" % k)
m = load(cfg, a.start, at)
r = plan(m, a.mode)
except (InputError, KeyError, TypeError, ValueError) as e:
print("error: %s" % e, file=sys.stderr)
return 2
if a.json:
out = {"books": [{"title": b["title"], "deadline": str(b["deadline"]) if b["deadline"] else None, "finish": str(b.get("finish")) if b.get("finish") else None,
"minutes_needed": round(b["need"])} for b in r["all_books"]],
"days": [{"date": str(x["date"]), "minutes": round(x["planned"]), "read": [fmt_item(it) for it in x["items"]]} for x in r["days"]],
"findings": [{"severity": s, "code": c, "message": msg} for s, c, msg in r["findings"]]}
print(json.dumps(out, indent=2))
else:
print(report(r, m, a.mode))
return 1 if any(f[0] == "high" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
---
name: invoice-line-item-and-vat-total-checker
description: Checks invoices before you pay or send them - recomputes every line (quantity x unit price minus discount) and the VAT or sales tax per line, the tax breakdown and the totals from a JSON or CSV export. Reports arithmetic errors, rates outside your list, per-line vs per-invoice rounding, discounts, duplicate lines and invoices, missing required fields and date problems. Includes an offline stdlib Python checker. Not tax advice.
---
# Invoice Line Item and VAT Total Checker
You help small businesses, freelancers and finance assistants catch invoice mistakes before money moves: a line total that does not equal quantity x price, a tax amount worked out on the wrong base, a rate that is not one you use, a delivery line taxed at a rate from another country, the same invoice entered twice, a credit note with no VAT ID. You recompute everything from the lines, compare with what the invoice says, and hand back a discrepancy report someone can act on.
You check arithmetic and consistency against the rules the user gives you. You do not decide which tax rate legally applies, whether tax is due or which fields the law requires; say so in every report and point to an accountant or the tax authority for those questions.
## Files in this skill
- `scripts/invoice_check.py` - loads an invoices JSON (header, lines, stated totals, optional tax breakdown) or a flat line-items CSV (one row per line, grouped by invoice number and seller); recomputes line net, tax and gross with Decimal arithmetic, per-line or per-invoice tax rounding, half-up or half-even, tax-exclusive or tax-inclusive prices, line and invoice discounts; reports findings IV01-IV13 and a per-invoice line table (Python 3.8+ standard library only)
- `references/checks-and-codes.md` - every finding code, its severity and what to do about it
- `references/tax-math-and-rounding.md` - the formulas, rounding methods, discounts, credit notes and tolerance
- `templates/discrepancy-report.md` - the report to send to the supplier, client or bookkeeper
- `examples/invoices.json` - five purchase invoices of a design studio: one clean, one with a wrong line, a non-listed rate, a duplicated line and a category rate mismatch, a cloud bill with fractional quantities and a due date before the invoice date, a duplicate entry and a credit note without a VAT ID
- `examples/vat-rules.json` - example rules: allowed rates 20/10/0, a category -> rate list, per-line rounding, required fields (replace the rates with yours)
- `examples/invoice-lines.csv` - two of the studio's own sales invoices as a line-items export, one with a tax error, a line left out of the totals, a missing buyer and a currency that changes mid-invoice
- `examples/invoice-report.md` - the real checker output on invoices.json
## Workflow
### 1. Collect the invoices and the rules
Ask for the invoices as an export (accounting tool CSV, e-invoice JSON) or paste the lines; one file can hold a whole batch. Then ask for the rules: the tax rates the business actually uses, any item categories with their own rate (from their accountant, not from you), whether prices are tax-exclusive or tax-inclusive, how the tool rounds tax (per line or per invoice), the currency decimals and which header fields their invoices must carry. Write them into a rules JSON like `examples/vat-rules.json`.
### 2. Run the checker
```
python3 scripts/invoice_check.py examples/invoices.json --rules examples/vat-rules.json --today 2026-11-03
python3 scripts/invoice_check.py examples/invoice-lines.csv --rules examples/vat-rules.json
python3 scripts/invoice_check.py batch.json --rules rules.json --json
python3 scripts/invoice_check.py --self-test
```
Exit 0 = no high findings, 1 = high findings, 2 = bad input. Without a shell, recompute by hand with `references/tax-math-and-rounding.md`: each line net = round(qty x unit price x (1 - discount)), tax = round(net x rate), then sum per rate and compare with the stated totals.
### 3. Read the result
- High first: IV02/IV03 (a line is wrong), IV05 (totals do not add up), IV04 (a rate not in your list), IV07 (same invoice number twice), IV01 (no number, date, quantity or price). Do not pay or send these.
- Medium: category rate mismatches, duplicate lines, negative lines on a normal invoice, due dates before the invoice date, missing VAT IDs or buyer names, a header that changes between CSV rows.
- Low/info: rounding-method differences (IV06) and extra decimals (IV12) - usually explainable, note them.
- A wrong total often comes from one wrong line: fix the line, then re-run before reporting the total.
### 4. Decide and write it up
For each invoice: OK to pay/send, ask for a corrected invoice or credit note, or check internally (duplicate entry, wrong rule). Fill `templates/discrepancy-report.md` with the line, the stated and recomputed amounts and the exact correction you need. Keep the tone factual: "line 1: 250 x 0.38 = 95.00, invoiced 98.00".
### 5. Deliver
Return the per-invoice verdicts, the discrepancy report(s), a list of questions for the accountant (rates, required fields) and any rule changes for next time (e.g. switch to per-invoice rounding if that is what the supplier's system does).
## Example result
On `examples/invoices.json` 2 of 5 invoices must not be paid as they are. Brightline Print 2026/0877 bills 250 business cards at 0.38 as 98.00 instead of 95.00 (IV02), taxes delivery at 18%, which is not in the studio's rates (IV04), repeats the flyer line (IV07) and taxes catering at 20% where the studio's list says 10% (IV04, medium); its totals are 3.00 net / 0.60 tax too high (IV05). Northwind's INV-2026-1041 was entered twice (IV07, high). CloudNest's tax of 7.38 is per-invoice rounding instead of 7.39 per line (IV06 info, within tolerance) but its due date is before the invoice date; the Brightline credit note has no seller VAT ID. The clean Northwind invoice passes.
## Rules
- Recompute from the lines; never trust a stated total, and report the first wrong line before the totals it breaks.
- Use only the rates and categories the user gives you. If asked which rate applies, say that is a tax question for their accountant or tax authority.
- Use exact decimal arithmetic and the user's rounding method; a 0.01 rounding difference is not an error unless their rules say so.
- Treat duplicates as "possible": show both invoices and let the user confirm.
- Never change an invoice yourself; ask the issuer for a corrected invoice or a credit note.
- Keep invoice data private: do not paste it into public tools or share it beyond the people who need it.
FILE:references/checks-and-codes.md
# Checks and finding codes
| Code | Severity | Check | Typical fix |
|---|---|---|---|
| IV01 | high: number, date, qty, price, no lines, no totals; medium: other fields | required invoice and line fields present and parseable (fields come from `required_invoice_fields` / `required_line_fields`) | ask for a corrected invoice with the missing field |
| IV02 | high | line net = round(qty x unit price x (1 - discount%)); line gross = net + tax | corrected invoice for that line |
| IV03 | high | line tax = round(line net x rate); a tax that is consistent with an already wrong net is not reported twice | corrected invoice |
| IV04 | high: rate not in `allowed_vat_rates`; medium: rate differs from your `category_rates`; low: rate written as a fraction (0.2) | rate sanity against YOUR list | check the rate with your accountant, then ask the issuer |
| IV05 | high (medium if within tolerance x lines) | stated net, tax and grand total vs the recomputed sums; grand total = net + tax | corrected invoice; often follows from IV02/IV03 |
| IV06 | low (info if within tolerance) | tax total matches the other rounding method (per invoice vs per line) | accept, or set `vat_rounding` to what the issuer uses |
| IV07 | high: same seller + invoice number twice; medium: same seller, date and total under another number, or a repeated line | duplicates | confirm with the issuer before paying twice |
| IV08 | high: discount outside 0-100%; info: invoice-level discount applied | discounts reduce the taxable base before tax | corrected invoice |
| IV09 | medium: negative qty/price on a normal invoice; low: zero qty/price | returns belong on a credit note (`"type": "credit_note"`) | ask for a credit note |
| IV10 | high: invalid invoice date; medium: invalid or earlier due date, invoice date in the future | dates | corrected invoice / check the entry |
| IV11 | medium | stated tax breakdown per rate vs the lines | corrected invoice |
| IV12 | low | stated amounts with more decimals than `currency_decimals` (unit prices may have more) | usually cosmetic |
| IV13 | medium | CSV export: header values (currency, buyer, date...) differ between rows of one invoice | fix the export or the source record |
## Verdicts
- **FIX**: at least one high finding. Do not pay or send.
- **CHECK**: medium findings only. Usually a question for the issuer or an internal check.
- **OK**: nothing, or only low/info.
## Input fields
JSON invoice: `number, date, due_date, type (invoice|credit_note), seller{name, vat_id}, buyer{name, vat_id}, currency, invoice_discount_pct, lines[], totals{net, vat, gross}, vat_breakdown[{rate, net, vat}]`.
JSON line: `sku, description, category, qty, unit_price, discount_pct, vat_rate, net, vat, gross` (stated amounts are optional - lines without them are only recomputed).
CSV columns: `invoice_number, invoice_date, due_date, type, seller_name, seller_vat_id, buyer_name, buyer_vat_id, currency, invoice_discount_pct, sku, description, category, qty, unit_price, discount_pct, vat_rate, line_net, vat_amount, line_gross, total_net, total_vat, total_gross` (totals on any row of the invoice). Decimal commas (12,50 and 1.234,56) are accepted.
FILE:references/tax-math-and-rounding.md
# Tax math and rounding
This is arithmetic, not tax law. Rates, exemptions, reverse charge and required invoice fields depend on the country and the transaction: take them from the user's accountant or tax authority.
## Tax-exclusive prices (default)
```
line net = round(qty x unit_price x (1 - discount_pct / 100))
line tax = round(line net x rate / 100) per-line rounding
line gross = line net + line tax
net total = sum(line net) (x (1 - invoice_discount_pct / 100) per rate, rounded)
tax total = sum(line tax) per-line rounding
= sum over rates of round(net of that rate x rate / 100) per-invoice rounding
grand total = net total + tax total
```
## Tax-inclusive prices (`prices_include_vat: true`)
```
line gross = round(qty x unit_price x (1 - discount_pct / 100))
line tax = round(line gross x rate / (100 + rate))
line net = line gross - line tax
```
Example: gross 12.00 at 20% -> tax 2.00, net 10.00.
## Rounding
- `half_up` (default): 0.005 -> 0.01. Most invoicing tools and many tax authorities use it.
- `half_even` (banker's rounding): 0.005 -> 0.00, 0.015 -> 0.02. Some ERP and spreadsheet functions use it.
- Per-line vs per-invoice rounding can differ by up to 0.005 x number of lines. Ten lines of 0.05 at 10%: per line 10 x 0.01 = 0.10, per invoice round(0.50 x 10%) = 0.05. Both can be legitimate; check which method the issuer uses and set `vat_rounding`.
- `tolerance` (default 0.01) is the difference accepted per amount; differences at or below it are not errors (IV06 info shows when they come from the rounding method).
## Discounts
- Line discounts reduce the line net before tax.
- An invoice-level discount (`invoice_discount_pct`) reduces the taxable base of every rate proportionally before tax; the checker applies it per rate.
- Early-payment (settlement) discounts that are only taken later are not modelled; check them separately.
## Credit notes and negative lines
- A credit note (`"type": "credit_note"`) has negative quantities or prices and negative totals; tax is negative too.
- A negative line on a normal invoice (a return netted off) is flagged (IV09) because many systems and auditors expect a separate credit note.
## Multi-currency
- All amounts of one invoice must be in one currency; a currency that changes between CSV rows is flagged (IV13).
- Conversion to your book currency is out of scope: convert after the check with the rate your bookkeeping uses.
FILE:templates/discrepancy-report.md
# Invoice discrepancy report
**To:** <supplier / client / bookkeeper> **From:** <name> **Date:** <YYYY-MM-DD>
**Invoice:** <number> dated <date>, total <stated total> <currency>
We checked this invoice line by line before <paying / sending> it and found the following. Please send a corrected invoice or a credit note for the difference.
| # | Line / field | Invoiced | Should be | Difference | Note |
|---|---|---|---|---|---|
| 1 | <line 1: 250 x 0.38> | <98.00> | <95.00> | <3.00> | <quantity x unit price> |
| 2 | <tax rate on delivery> | <18%> | <rate we expect, confirmed by our accountant> | <...> | <...> |
**Totals**
| | Invoiced | Recomputed |
|---|---|---|
| Net | <...> | <...> |
| Tax | <...> | <...> |
| Total | <...> | <...> |
**Other points**
- <duplicate line / missing VAT ID / due date before invoice date>
**What we need:** <corrected invoice | credit note for X | confirmation that invoice Y is a duplicate>
*Recomputed from the invoice lines with <rounding> rounding, tolerance <0.01>. This is an arithmetic check, not tax advice.*
FILE:examples/invoices.json
{
"invoices": [
{
"number": "INV-2026-1041",
"date": "2026-10-05",
"due_date": "2026-11-04",
"seller": {"name": "Northwind Office Supplies", "vat_id": "GB123456789"},
"buyer": {"name": "Lumen Studio Ltd"},
"currency": "GBP",
"lines": [
{"sku": "PAP-A4-5", "description": "A4 copy paper, box of 5 reams", "qty": 4, "unit_price": "21.50", "vat_rate": 20, "net": "86.00", "vat": "17.20", "gross": "103.20"},
{"sku": "TON-HP-26X", "description": "Toner cartridge 26X", "qty": 2, "unit_price": "89.99", "discount_pct": 5, "vat_rate": 20, "net": "170.98", "vat": "34.20", "gross": "205.18"},
{"sku": "BK-TYPO", "description": "Book: Thinking with Type", "category": "books", "qty": 1, "unit_price": "24.00", "vat_rate": 0, "net": "24.00", "vat": "0.00", "gross": "24.00"}
],
"totals": {"net": "280.98", "vat": "51.40", "gross": "332.38"},
"vat_breakdown": [{"rate": 20, "net": "256.98", "vat": "51.40"}, {"rate": 0, "net": "24.00", "vat": "0.00"}]
},
{
"number": "2026/0877",
"date": "2026-10-12",
"due_date": "2026-10-26",
"seller": {"name": "Brightline Print", "vat_id": "GB987654321"},
"buyer": {"name": "Lumen Studio Ltd"},
"currency": "GBP",
"lines": [
{"sku": "BC-350", "description": "Business cards, 350gsm", "qty": 250, "unit_price": "0.38", "vat_rate": 20, "net": "98.00", "vat": "19.60", "gross": "117.60"},
{"sku": "FLY-A5", "description": "A5 flyers, double-sided", "qty": 1000, "unit_price": "0.12", "discount_pct": 10, "vat_rate": 20, "net": "108.00", "vat": "21.60", "gross": "129.60"},
{"sku": "FLY-A5", "description": "A5 flyers, double-sided", "qty": 1000, "unit_price": "0.12", "discount_pct": 10, "vat_rate": 20, "net": "108.00", "vat": "21.60", "gross": "129.60"},
{"sku": "DEL-STD", "description": "Delivery", "qty": 1, "unit_price": "12.50", "vat_rate": 18, "net": "12.50", "vat": "2.25", "gross": "14.75"},
{"sku": "CAT-01", "description": "Catering for launch event", "category": "catering", "qty": 1, "unit_price": "150.00", "vat_rate": 20, "net": "150.00", "vat": "30.00", "gross": "180.00"}
],
"totals": {"net": "476.50", "vat": "95.05", "gross": "571.55"}
},
{
"number": "CN-8841",
"date": "2026-10-31",
"due_date": "2026-10-01",
"seller": {"name": "CloudNest Hosting", "vat_id": "IE6388047V"},
"buyer": {"name": "Lumen Studio Ltd"},
"currency": "GBP",
"lines": [
{"description": "Object storage, GB-month", "qty": "412.5", "unit_price": "0.0185", "vat_rate": 20},
{"description": "Egress, GB", "qty": "96.3", "unit_price": "0.0725", "vat_rate": 20},
{"description": "Load balancer hours", "qty": 744, "unit_price": "0.0225", "vat_rate": 20},
{"description": "Snapshot storage, GB-month", "qty": "38.4", "unit_price": "0.0420", "vat_rate": 20},
{"description": "DNS queries, per million", "qty": "1.7", "unit_price": "0.35", "vat_rate": 20},
{"description": "Static IP hours", "qty": 744, "unit_price": "0.0045", "vat_rate": 20}
],
"totals": {"net": "36.91", "vat": "7.38", "gross": "44.29"}
},
{
"number": "INV-2026-1041",
"date": "2026-10-05",
"seller": {"name": "Northwind Office Supplies", "vat_id": "GB123456789"},
"buyer": {"name": "Lumen Studio Ltd"},
"currency": "GBP",
"lines": [
{"sku": "PAP-A4-5", "description": "A4 copy paper, box of 5 reams", "qty": 4, "unit_price": "21.50", "vat_rate": 20},
{"sku": "TON-HP-26X", "description": "Toner cartridge 26X", "qty": 2, "unit_price": "89.99", "discount_pct": 5, "vat_rate": 20},
{"sku": "BK-TYPO", "description": "Book: Thinking with Type", "category": "books", "qty": 1, "unit_price": "24.00", "vat_rate": 0}
],
"totals": {"net": "280.98", "vat": "51.40", "gross": "332.38"}
},
{
"number": "CR-0012",
"type": "credit_note",
"date": "2026-10-20",
"seller": {"name": "Brightline Print"},
"buyer": {"name": "Lumen Studio Ltd"},
"currency": "GBP",
"lines": [
{"sku": "BC-350", "description": "Credit: misprinted business cards", "qty": -50, "unit_price": "0.38", "vat_rate": 20, "net": "-19.00", "vat": "-3.80", "gross": "-22.80"}
],
"totals": {"net": "-19.00", "vat": "-3.80", "gross": "-22.80"}
}
]
}
FILE:examples/vat-rules.json
{
"name": "Example rules for a studio's purchase invoices - replace the rates with the ones that apply to you",
"currency_decimals": 2,
"rounding": "half_up",
"vat_rounding": "line",
"tolerance": "0.01",
"prices_include_vat": false,
"allowed_vat_rates": [20, 10, 0],
"category_rates": {"books": 0, "catering": 10, "food": 10},
"required_invoice_fields": ["number", "date", "seller.name", "seller.vat_id", "buyer.name", "currency"],
"required_line_fields": ["description", "qty", "unit_price", "vat_rate"],
"max_discount_pct": 100
}
FILE:examples/invoice-lines.csv
invoice_number,invoice_date,due_date,seller_name,seller_vat_id,buyer_name,currency,sku,description,category,qty,unit_price,discount_pct,vat_rate,line_net,vat_amount,line_gross,total_net,total_vat,total_gross
S-1001,2026-10-28,2026-11-27,Lumen Studio Ltd,GB555000111,Harbour Cafe,GBP,LOGO,"Logo design, 3 concepts",,1,850.00,,20,850.00,170.00,1020.00,1170.00,234.00,1404.00
S-1001,2026-10-28,2026-11-27,Lumen Studio Ltd,GB555000111,Harbour Cafe,GBP,MENU,Menu layout (A4 folded),,2,160.00,,20,320.00,64.00,384.00,,,
S-1002,2026-10-30,2026-11-29,Lumen Studio Ltd,GB555000111,,GBP,WEB-DAY,Website build - day rate,,4.5,480.00,10,20,1944.00,388.80,2332.80,2104.00,420.80,2524.80
S-1002,2026-10-30,2026-11-29,Lumen Studio Ltd,GB555000111,,GBP,PHOTO,"Stock photo licence, 5 images",,5,32.00,,20,160.00,23.00,183.00,,,
S-1002,2026-10-30,2026-11-29,Lumen Studio Ltd,GB555000111,,EUR,BOOK,Brand guidelines book (printed),books,1,45.00,,0,45.00,0.00,45.00,,,
FILE:examples/invoice-report.md
# Invoice check: examples/invoices.json
5 invoice(s), 18 line(s). Rules: half up rounding, tax rounded per line, tolerance 0.01, prices exclude tax, allowed rates 20/10/0%.
**Verdict: 2 of 5 invoice(s) have high findings - do not pay/send until fixed**
Findings: 5 high, 4 medium, 0 low, 1 info.
This is an arithmetic and consistency check against your own rules, not tax advice: confirm rates and required invoice fields with your accountant or tax authority.
## Invoices
| Invoice | Seller | Lines | Stated total | Recomputed | Difference | High | Medium | Low | Status |
|---|---|---|---|---|---|---|---|---|---|
| INV-2026-1041 | Northwind Office Supplies | 3 | 332.38 GBP | 332.38 | 0.00 | 0 | 0 | 0 | OK |
| 2026/0877 | Brightline Print | 5 | 571.55 GBP | 567.95 | 3.60 | 4 | 2 | 0 | FIX |
| CN-8841 | CloudNest Hosting | 6 | 44.29 GBP | 44.30 | -0.01 | 0 | 1 | 0 | CHECK |
| INV-2026-1041 (#4) | Northwind Office Supplies | 3 | 332.38 GBP | 332.38 | 0.00 | 1 | 0 | 0 | FIX |
| CR-0012 (credit note) | Brightline Print | 1 | -22.80 GBP | -22.80 | 0.00 | 0 | 1 | 0 | CHECK |
## 2026/0877 - Brightline Print
| # | Item | Qty | Unit price | Disc % | Rate % | Net (stated / calc) | Tax (stated / calc) | Issues |
|---|---|---|---|---|---|---|---|---|
| 1 | Business cards, 350gsm | 250 | 0.38 | | 20 | 98.00 / 95.00 | 19.60 / 19.00 | net |
| 2 | A5 flyers, double-sided | 1000 | 0.12 | 10 | 20 | 108.00 / 108.00 | 21.60 / 21.60 | ok |
| 3 | A5 flyers, double-sided | 1000 | 0.12 | 10 | 20 | 108.00 / 108.00 | 21.60 / 21.60 | duplicate |
| 4 | Delivery | 1 | 12.50 | | 18 | 12.50 / 12.50 | 2.25 / 2.25 | rate |
| 5 | Catering for launch event | 1 | 150.00 | | 20 | 150.00 / 150.00 | 30.00 / 30.00 | category rate |
| Totals | Stated | Recomputed |
|---|---|---|
| Net | 476.50 | 473.50 |
| Tax | 95.05 | 94.45 |
| Total | 571.55 | 567.95 |
| Tax by rate | | 20%: 92.20 on 461.00; 18%: 2.25 on 12.50 |
- **HIGH IV02** line 1 (Business cards, 350gsm): line net 98.00 but 250 x 0.38 = 95.00 (diff 3.00)
- **HIGH IV04** line 4 (Delivery): tax rate 18% is not in your allowed rates (0, 10, 20)
- **HIGH IV05** stated net total 476.50 but the lines add up to 473.50 (diff 3.00)
- **HIGH IV05** stated tax total 95.05 but the lines add up to 94.45 (diff 0.60)
- **MEDIUM IV04** line 5 (Catering for launch event): category 'catering' is 10% in your rate list but the line uses 20%
- **MEDIUM IV07** line 3 (A5 flyers, double-sided) repeats line 2 (same item, quantity and price) - duplicate?
## CN-8841 - CloudNest Hosting
| # | Item | Qty | Unit price | Disc % | Rate % | Net (stated / calc) | Tax (stated / calc) | Issues |
|---|---|---|---|---|---|---|---|---|
| 1 | Object storage, GB-month | 412.5 | 0.0185 | | 20 | - / 7.63 | - / 1.53 | ok |
| 2 | Egress, GB | 96.3 | 0.0725 | | 20 | - / 6.98 | - / 1.40 | ok |
| 3 | Load balancer hours | 744 | 0.0225 | | 20 | - / 16.74 | - / 3.35 | ok |
| 4 | Snapshot storage, GB-month | 38.4 | 0.0420 | | 20 | - / 1.61 | - / 0.32 | ok |
| 5 | DNS queries, per million | 1.7 | 0.35 | | 20 | - / 0.60 | - / 0.12 | ok |
| 6 | Static IP hours | 744 | 0.0045 | | 20 | - / 3.35 | - / 0.67 | ok |
| Totals | Stated | Recomputed |
|---|---|---|
| Net | 36.91 | 36.91 |
| Tax | 7.38 | 7.39 |
| Total | 44.29 | 44.30 |
- **MEDIUM IV10** due date 2026-10-01 is before the invoice date 2026-10-31
- **INFO IV06** tax total 7.38 is per-invoice rounding (line rounding gives 7.39) - within tolerance
## INV-2026-1041 (#4) - Northwind Office Supplies
| # | Item | Qty | Unit price | Disc % | Rate % | Net (stated / calc) | Tax (stated / calc) | Issues |
|---|---|---|---|---|---|---|---|---|
| 1 | A4 copy paper, box of 5 reams | 4 | 21.50 | | 20 | - / 86.00 | - / 17.20 | ok |
| 2 | Toner cartridge 26X | 2 | 89.99 | 5 | 20 | - / 170.98 | - / 34.20 | ok |
| 3 | Book: Thinking with Type | 1 | 24.00 | | 0 | - / 24.00 | - / 0.00 | ok |
| Totals | Stated | Recomputed |
|---|---|---|
| Net | 280.98 | 280.98 |
| Tax | 51.40 | 51.40 |
| Total | 332.38 | 332.38 |
| Tax by rate | | 20%: 51.40 on 256.98; 0%: 0.00 on 24.00 |
- **HIGH IV07** invoice number INV-2026-1041 from this seller already appears (invoice #1) - duplicate invoice?
## CR-0012 - Brightline Print
| # | Item | Qty | Unit price | Disc % | Rate % | Net (stated / calc) | Tax (stated / calc) | Issues |
|---|---|---|---|---|---|---|---|---|
| 1 | Credit: misprinted business cards | -50 | 0.38 | | 20 | -19.00 / -19.00 | -3.80 / -3.80 | ok |
| Totals | Stated | Recomputed |
|---|---|---|
| Net | -19.00 | -19.00 |
| Tax | -3.80 | -3.80 |
| Total | -22.80 | -22.80 |
- **MEDIUM IV01** missing invoice field 'seller.vat_id'
FILE:scripts/invoice_check.py
#!/usr/bin/env python3
"""Invoice line item and VAT / sales-tax total checker (stdlib only, Python 3.8+).
Recomputes every line (qty x unit price - discount, tax per line) and every invoice total from
an invoices JSON or a flat line-items CSV, compares them with the stated amounts and reports
arithmetic errors, tax rates outside your list, rounding-method differences, discounts,
duplicate lines and invoices, missing required fields and date problems.
It checks numbers against the rules YOU give it; it does not decide which tax rate applies.
Usage:
invoice_check.py INVOICES.json|LINES.csv [--rules RULES.json] [--today YYYY-MM-DD] [--json]
invoice_check.py --self-test
Exit: 0 = no high findings, 1 = high findings (wrong totals, wrong rate, missing number/date,
duplicate invoice), 2 = bad input.
"""
import argparse, csv, datetime, io, json, re, sys
from decimal import Decimal, ROUND_HALF_UP, ROUND_HALF_EVEN, InvalidOperation
DEFAULT_RULES = {
"currency_decimals": 2,
"rounding": "half_up", # half_up | half_even
"vat_rounding": "line", # line = round tax per line then sum; invoice = per rate on the total
"tolerance": "0.01", # accepted difference per amount
"prices_include_vat": False, # unit prices are gross (tax-inclusive)
"allowed_vat_rates": None, # e.g. [20, 10, 1, 0]; None = do not check
"category_rates": {}, # your own mapping, e.g. {"books": 0}
"required_invoice_fields": ["number", "date", "seller.name", "seller.vat_id", "buyer.name", "currency"],
"required_line_fields": ["description", "qty", "unit_price", "vat_rate"],
"max_discount_pct": 100,
}
SEV = {"high": 0, "medium": 1, "low": 2, "info": 3}
CSV_MAP = {"invoice_number": "number", "invoice_date": "date", "due_date": "due_date", "type": "type",
"seller_name": "seller.name", "seller_vat_id": "seller.vat_id", "buyer_name": "buyer.name",
"buyer_vat_id": "buyer.vat_id", "currency": "currency", "invoice_discount_pct": "invoice_discount_pct",
"total_net": "totals.net", "total_vat": "totals.vat", "total_gross": "totals.gross"}
LINE_MAP = {"sku": "sku", "description": "description", "category": "category", "qty": "qty",
"unit_price": "unit_price", "discount_pct": "discount_pct", "line_net": "net", "vat_rate": "vat_rate",
"vat_amount": "vat", "line_gross": "gross"}
class InputError(Exception):
pass
def D(v, what="number"):
if v is None or (isinstance(v, str) and not v.strip()):
return None
if isinstance(v, float):
v = repr(v)
try:
s = str(v).strip().replace(" ", "")
if re.match(r"^-?\d{1,3}(\.\d{3})+,\d+$", s): # 1.234,56
s = s.replace(".", "").replace(",", ".")
elif re.match(r"^-?\d+,\d+$", s): # 12,50
s = s.replace(",", ".")
else:
s = s.replace(",", "") # 1,234.56
s = s.rstrip("%")
return Decimal(s)
except (InvalidOperation, ValueError):
raise InputError("not a %s: %r" % (what, v))
def get(d, path):
for k in path.split("."):
if not isinstance(d, dict):
return None
d = d.get(k)
return d
def setp(d, path, v):
ks = path.split(".")
for k in ks[:-1]:
d = d.setdefault(k, {})
d[ks[-1]] = v
def blank(v):
return v is None or (isinstance(v, str) and not v.strip())
# ---------------------------------------------------------------- loading
def load_invoices(path_or_text, is_csv=None):
text = path_or_text
if is_csv is None:
is_csv = not text.lstrip().startswith(("{", "["))
if not is_csv:
try:
data = json.loads(text)
except ValueError as e:
raise InputError("invalid JSON: %s" % e)
invs = data.get("invoices") if isinstance(data, dict) else data
if not isinstance(invs, list):
raise InputError("JSON must be a list of invoices or {\"invoices\": [...]}")
rules = data.get("rules") if isinstance(data, dict) else None
return invs, rules, []
rows = list(csv.DictReader(io.StringIO(text.lstrip("\ufeff"))))
if not rows:
raise InputError("CSV has no rows")
cols = set(rows[0].keys())
if "invoice_number" not in cols or not ({"qty", "unit_price"} <= cols):
raise InputError("CSV needs at least invoice_number, qty and unit_price columns")
invs, order, notes = {}, [], []
for n, row in enumerate(rows, 2):
key = ((row.get("seller_vat_id") or row.get("seller_name") or "").strip(), (row.get("invoice_number") or "").strip())
if key not in invs:
inv = {"_rows": [n]}
for c, p in CSV_MAP.items():
if c in row and not blank(row[c]):
setp(inv, p, row[c].strip())
invs[key] = inv; order.append(key)
else:
inv = invs[key]; inv["_rows"].append(n)
for c, p in CSV_MAP.items():
if c in row and not blank(row[c]) and not blank(get(inv, p)) and str(get(inv, p)) != row[c].strip():
notes.append((inv, "row %d: %s is '%s' but earlier rows say '%s'" % (n, c, row[c].strip(), get(inv, p))))
elif c in row and not blank(row[c]) and blank(get(inv, p)):
setp(inv, p, row[c].strip())
line = {"_row": n}
for c, p in LINE_MAP.items():
if c in row and not blank(row[c]):
line[p] = row[c].strip()
inv.setdefault("lines", []).append(line)
return [invs[k] for k in order], None, notes
def parse_date(s):
try:
return datetime.date.fromisoformat(str(s)[:10]) if not blank(s) else None
except ValueError:
return "bad"
# ---------------------------------------------------------------- checking
def check(invoices, rules, today=None, csv_notes=()):
R = dict(DEFAULT_RULES); R.update(rules or {})
q = Decimal(1).scaleb(-int(R["currency_decimals"]))
mode = ROUND_HALF_EVEN if R["rounding"] == "half_even" else ROUND_HALF_UP
tol = Decimal(str(R["tolerance"]))
rnd = lambda x: x.quantize(q, rounding=mode)
allowed = None if R["allowed_vat_rates"] is None else {Decimal(str(r)) for r in R["allowed_vat_rates"]}
catmap = {k.lower(): Decimal(str(v)) for k, v in (R.get("category_rates") or {}).items()}
results, seen_numbers, seen_amounts = [], {}, {}
notes_by = {}
for inv, msg in csv_notes:
notes_by.setdefault(id(inv), []).append(msg)
for idx, inv in enumerate(invoices):
F = []
def f(sev, code, msg):
F.append((sev, code, msg))
num = inv.get("number")
label = str(num) if not blank(num) else "#%d (no number)" % (idx + 1)
if any(str(o.get("number")) == str(num) for o in invoices[:idx]) and not blank(num):
label += " (#%d)" % (idx + 1)
credit = str(inv.get("type", "")).lower().replace(" ", "_") in ("credit_note", "credit", "refund")
for p in R["required_invoice_fields"]:
if blank(get(inv, p)):
f("high" if p in ("number", "date") else "medium", "IV01", "missing invoice field '%s'" % p)
for m in notes_by.get(id(inv), []):
f("medium", "IV13", "inconsistent header across CSV rows: " + m)
d = parse_date(inv.get("date")); dd = parse_date(inv.get("due_date"))
if d == "bad":
f("high", "IV10", "invoice date '%s' is not YYYY-MM-DD" % inv.get("date")); d = None
if dd == "bad":
f("medium", "IV10", "due date '%s' is not YYYY-MM-DD" % inv.get("due_date")); dd = None
if d and dd and dd < d:
f("medium", "IV10", "due date %s is before the invoice date %s" % (dd, d))
if d and today and d > today:
f("medium", "IV10", "invoice date %s is in the future (today %s)" % (d, today))
lines, per_rate, sigs = [], {}, {}
sum_net = sum_vat_line = Decimal(0)
for ln_i, ln in enumerate(inv.get("lines") or [], 1):
L = {"n": ln_i, "desc": str(ln.get("description") or ln.get("sku") or "")[:40], "issues": []}
where = "line %d (%s)" % (ln_i, L["desc"] or "no description")
for p in R["required_line_fields"]:
if blank(ln.get(p)):
f("high" if p in ("qty", "unit_price") else "medium", "IV01", "%s: missing '%s'" % (where, p)); L["issues"].append("missing " + p)
try:
qty = D(ln.get("qty"), "quantity"); up = D(ln.get("unit_price"), "unit price")
disc = D(ln.get("discount_pct"), "discount") or Decimal(0)
rate = D(ln.get("vat_rate"), "tax rate")
s_net, s_vat, s_gross = D(ln.get("net")), D(ln.get("vat")), D(ln.get("gross"))
except InputError as e:
f("high", "IV01", "%s: %s" % (where, e)); L["issues"].append("unparseable"); lines.append(L); continue
if rate is not None and Decimal(0) < rate < 1:
f("low", "IV04", "%s: tax rate %s looks like a fraction - read as %s%%" % (where, rate, rate * 100)); rate = rate * 100
if qty is None or up is None:
lines.append(L); continue
if disc < 0 or disc > Decimal(str(R["max_discount_pct"])):
f("high", "IV08", "%s: discount %s%% is outside 0-%s%%" % (where, disc, R["max_discount_pct"])); L["issues"].append("discount")
if (qty < 0 or up < 0) and not credit:
f("medium", "IV09", "%s: negative quantity or price on an invoice - should this be a credit note?" % where); L["issues"].append("negative")
if qty == 0 or up == 0:
f("low", "IV09", "%s: zero quantity or price" % where)
r = rate if rate is not None else Decimal(0)
base = qty * up * (1 - disc / 100)
if R["prices_include_vat"]:
gross = rnd(base); vat = rnd(gross * r / (100 + r)); net = gross - vat
else:
net = rnd(base); vat = rnd(net * r / 100); gross = net + vat
L.update(qty=qty, up=up, disc=disc, rate=rate, net=net, vat=vat, gross=gross, s_net=s_net, s_vat=s_vat, s_gross=s_gross)
if rate is not None:
if allowed is not None and rate not in allowed:
f("high", "IV04", "%s: tax rate %s%% is not in your allowed rates (%s)" % (where, plain(rate), ", ".join(plain(x) for x in sorted(allowed))))
L["issues"].append("rate")
cat = str(ln.get("category") or "").lower()
if cat in catmap and catmap[cat] != rate:
f("medium", "IV04", "%s: category '%s' is %s%% in your rate list but the line uses %s%%" % (where, cat, plain(catmap[cat]), plain(rate)))
L["issues"].append("category rate")
if s_net is not None and abs(s_net - net) > tol:
f("high", "IV02", "%s: line net %s but %s x %s%s = %s (diff %s)" % (where, s_net, plain(qty), up, " - %s%%" % plain(disc) if disc else "", net, s_net - net))
L["issues"].append("net")
if s_vat is not None and abs(s_vat - vat) > tol:
exp_on_stated = rnd((s_net if s_net is not None else net) * r / 100)
if s_net is not None and abs(s_vat - exp_on_stated) <= tol:
pass # consistent with its own (wrong) net; already reported as IV02
else:
f("high", "IV03", "%s: tax %s but %s%% of %s = %s (diff %s)" % (where, s_vat, plain(r), net, vat, s_vat - vat))
L["issues"].append("tax")
if s_gross is not None and s_net is not None and s_vat is not None and abs(s_gross - (s_net + s_vat)) > tol:
f("high", "IV02", "%s: line gross %s is not net %s + tax %s" % (where, s_gross, s_net, s_vat)); L["issues"].append("gross")
for v, nm in ((s_net, "net"), (s_vat, "tax"), (s_gross, "gross")):
if v is not None and v != v.quantize(q, rounding=mode):
f("low", "IV12", "%s: stated %s %s has more than %s decimals" % (where, nm, v, R["currency_decimals"]))
sig = (str(ln.get("sku") or "").lower(), L["desc"].lower(), qty, up)
if sig in sigs:
f("medium", "IV07", "%s repeats line %d (same item, quantity and price) - duplicate?" % (where, sigs[sig])); L["issues"].append("duplicate")
else:
sigs[sig] = ln_i
pr = per_rate.setdefault(r, [Decimal(0), Decimal(0)])
pr[0] += net; pr[1] += vat
sum_net += net; sum_vat_line += vat
lines.append(L)
if not lines:
f("high", "IV01", "invoice has no lines")
# invoice-level discount reduces the taxable base proportionally
idisc = D(inv.get("invoice_discount_pct"), "invoice discount") if not blank(inv.get("invoice_discount_pct")) else None
if idisc:
if idisc < 0 or idisc > 100:
f("high", "IV08", "invoice discount %s%% is outside 0-100%%" % idisc)
factor = 1 - idisc / 100
for r in per_rate:
n_r = rnd(per_rate[r][0] * factor)
per_rate[r] = [n_r, rnd(n_r * r / 100)]
sum_net = sum(v[0] for v in per_rate.values())
sum_vat_line = sum(v[1] for v in per_rate.values())
f("info", "IV08", "invoice discount %s%% applied before tax (taxable base reduced on every rate)" % idisc)
vat_invoice_mode = sum((rnd(v[0] * r / 100) for r, v in per_rate.items()), Decimal(0))
exp_vat = sum_vat_line if R["vat_rounding"] == "line" else vat_invoice_mode
alt_vat = vat_invoice_mode if R["vat_rounding"] == "line" else sum_vat_line
exp = {"net": sum_net, "vat": exp_vat, "gross": sum_net + exp_vat}
if R["prices_include_vat"] and not idisc:
exp["gross"] = sum(L.get("gross", 0) for L in lines)
exp["net"] = exp["gross"] - exp_vat
T = inv.get("totals") or {}
try:
st = {k: D(T.get(k), "total") for k in ("net", "vat", "gross")}
except InputError as e:
f("high", "IV05", str(e)); st = {"net": None, "vat": None, "gross": None}
if all(v is None for v in st.values()):
f("high", "IV01", "no stated totals (net / tax / gross) to check")
for k, nm in (("net", "net total"), ("vat", "tax total"), ("gross", "grand total")):
if st[k] is None:
continue
diff = st[k] - exp[k]
if abs(diff) <= tol:
if k == "vat" and diff and st[k] == alt_vat:
f("info", "IV06", "tax total %s is %s rounding (%s rounding gives %s) - within tolerance" % (
st[k], "per-invoice" if R["vat_rounding"] == "line" else "per-line", R["vat_rounding"], exp_vat))
continue
if k == "vat" and abs(st[k] - alt_vat) <= tol:
f("low", "IV06", "tax total %s matches %s rounding (%s) instead of your %s rounding (%s) - a rounding-method difference of %s" % (
st[k], "per-invoice" if R["vat_rounding"] == "line" else "per-line", alt_vat, R["vat_rounding"], exp_vat, diff))
exp["vat"] = alt_vat; exp["gross"] = exp["net"] + alt_vat
continue
if k == "gross" and st["net"] is not None and st["vat"] is not None and abs(st["gross"] - (st["net"] + st["vat"])) <= tol and (abs(st["net"] - exp["net"]) > tol or abs(st["vat"] - exp["vat"]) > tol):
continue # follows from the net/tax error already reported
sev = "high" if abs(diff) > tol * max(1, len(lines)) else "medium"
f(sev, "IV05", "stated %s %s but the lines add up to %s (diff %s)" % (nm, st[k], exp[k], diff))
if st["net"] is not None and st["vat"] is not None and st["gross"] is not None and abs(st["gross"] - st["net"] - st["vat"]) > tol:
f("high", "IV05", "stated grand total %s is not stated net %s + tax %s" % (st["gross"], st["net"], st["vat"]))
for b in inv.get("vat_breakdown") or []:
try:
br, bn, bv = D(b.get("rate")), D(b.get("net")), D(b.get("vat"))
except InputError as e:
f("medium", "IV11", "tax breakdown: %s" % e); continue
got = per_rate.get(br)
if got is None:
f("medium", "IV11", "tax breakdown lists %s%% but no line uses that rate" % br)
elif (bn is not None and abs(bn - got[0]) > tol) or (bv is not None and abs(bv - got[1]) > tol and abs(bv - rnd(got[0] * br / 100)) > tol):
f("medium", "IV11", "tax breakdown %s%%: stated base %s / tax %s, lines give %s / %s" % (br, bn, bv, got[0], got[1]))
seller = str(get(inv, "seller.vat_id") or get(inv, "seller.name") or "").strip().lower()
if not blank(num):
k = (seller, str(num).strip().lower())
if k in seen_numbers:
f("high", "IV07", "invoice number %s from this seller already appears (invoice #%d) - duplicate invoice?" % (num, seen_numbers[k] + 1))
else:
seen_numbers[k] = idx
if st["gross"] is not None and d:
k = (seller, d, st["gross"])
if k in seen_amounts and str(invoices[seen_amounts[k]].get("number")) != str(num):
f("medium", "IV07", "same seller, date and total %s as invoice %s - possible duplicate under a new number" % (st["gross"], invoices[seen_amounts[k]].get("number")))
seen_amounts.setdefault(k, idx)
F.sort(key=lambda x: (SEV[x[0]], x[1]))
results.append({"label": label, "seller": get(inv, "seller.name") or "", "currency": inv.get("currency") or "",
"credit": credit, "lines": lines, "per_rate": per_rate, "expected": exp, "stated": st,
"findings": F, "rows": inv.get("_rows")})
return results, R
# ---------------------------------------------------------------- rendering
def plain(v):
"""Plain decimal without exponent or trailing zeros (250, 0.0185, 12.5)."""
if v is None:
return "-"
t = format(v.normalize(), "f")
return t
def money(v):
return "-" if v is None else "{:,.2f}".format(v)
def report(results, R, src):
tot = {s: sum(1 for r in results for f in r["findings"] if f[0] == s) for s in SEV}
bad = [r for r in results if any(f[0] == "high" for f in r["findings"])]
out = ["# Invoice check: %s" % src, ""]
out.append("%d invoice(s), %d line(s). Rules: %s rounding, tax rounded per %s, tolerance %s, prices %s tax%s." % (
len(results), sum(len(r["lines"]) for r in results), R["rounding"].replace("_", " "), R["vat_rounding"], R["tolerance"],
"include" if R["prices_include_vat"] else "exclude",
", allowed rates " + "/".join(str(x) for x in R["allowed_vat_rates"]) + "%" if R["allowed_vat_rates"] is not None else ""))
out += ["", "**Verdict: %s**" % ("%d of %d invoice(s) have high findings - do not pay/send until fixed" % (len(bad), len(results)) if bad else "no high findings"),
"", "Findings: %d high, %d medium, %d low, %d info." % (tot["high"], tot["medium"], tot["low"], tot["info"]), "",
"This is an arithmetic and consistency check against your own rules, not tax advice: confirm rates and required invoice fields with your accountant or tax authority.", "",
"## Invoices", "", "| Invoice | Seller | Lines | Stated total | Recomputed | Difference | High | Medium | Low | Status |", "|---|---|---|---|---|---|---|---|---|---|"]
for r in results:
c = {s: sum(1 for f in r["findings"] if f[0] == s) for s in SEV}
sg, eg = r["stated"]["gross"], r["expected"]["gross"]
out.append("| %s%s | %s | %d | %s %s | %s | %s | %d | %d | %d | %s |" % (
r["label"], " (credit note)" if r["credit"] else "", r["seller"], len(r["lines"]), money(sg), r["currency"], money(eg),
money(sg - eg) if sg is not None else "-", c["high"], c["medium"], c["low"],
"FIX" if c["high"] else ("CHECK" if c["medium"] else "OK")))
for r in results:
if not r["findings"] or all(f[0] == "info" for f in r["findings"]):
continue
out += ["", "## %s - %s" % (r["label"], r["seller"]), ""]
out += ["| # | Item | Qty | Unit price | Disc % | Rate % | Net (stated / calc) | Tax (stated / calc) | Issues |", "|---|---|---|---|---|---|---|---|---|"]
for L in r["lines"]:
if "net" not in L:
out.append("| %d | %s | - | - | - | - | - | - | %s |" % (L["n"], L["desc"], ", ".join(L["issues"])))
continue
out.append("| %d | %s | %s | %s | %s | %s | %s / %s | %s / %s | %s |" % (
L["n"], L["desc"], plain(L["qty"]), L["up"], plain(L["disc"]) if L["disc"] else "", plain(L["rate"]),
money(L["s_net"]), money(L["net"]), money(L["s_vat"]), money(L["vat"]), ", ".join(L["issues"]) or "ok"))
out += ["", "| Totals | Stated | Recomputed |", "|---|---|---|"]
for k, nm in (("net", "Net"), ("vat", "Tax"), ("gross", "Total")):
out.append("| %s | %s | %s |" % (nm, money(r["stated"][k]), money(r["expected"][k])))
if len(r["per_rate"]) > 1:
out.append("| Tax by rate | | %s |" % "; ".join("%s%%: %s on %s" % (plain(k), money(v[1]), money(v[0])) for k, v in sorted(r["per_rate"].items(), reverse=True)))
out += [""] + ["- **%s %s** %s" % (s.upper(), c, m) for s, c, m in r["findings"]]
out.append("")
return "\n".join(out)
# ---------------------------------------------------------------- self-test
def self_test():
tests = []
def t(name, cond):
tests.append((name, bool(cond)))
def inv(lines, totals, **kw):
base = {"number": "A1", "date": "2026-10-01", "seller": {"name": "S", "vat_id": "X1"}, "buyer": {"name": "B"},
"currency": "EUR", "lines": lines, "totals": totals}
base.update(kw); return base
codes = lambda res: {f[1] for f in res[0]["findings"]}
sev = lambda res, c: {f[0] for f in res[0]["findings"] if f[1] == c}
ok = [{"description": "a", "qty": 2, "unit_price": "10.00", "vat_rate": 20, "net": "20.00", "vat": "4.00", "gross": "24.00"}]
res, _ = check([inv(ok, {"net": "20.00", "vat": "4.00", "gross": "24.00"})], {})
t("clean invoice has no findings", not res[0]["findings"])
res, _ = check([inv([dict(ok[0], net="22.00", vat="4.40", gross="26.40")], {"net": "22.00", "vat": "4.40", "gross": "26.40"})], {})
t("wrong line net -> IV02 high", "high" in sev(res, "IV02"))
t("tax consistent with wrong net not double-reported", "IV03" not in codes(res))
res, _ = check([inv([dict(ok[0], vat="3.60", gross="23.60")], {"net": "20.00", "vat": "3.60", "gross": "23.60"})], {})
t("wrong line tax -> IV03", "IV03" in codes(res))
res, _ = check([inv(ok, {"net": "20.00", "vat": "4.00", "gross": "24.00"})], {"allowed_vat_rates": [10, 0]})
t("rate not allowed -> IV04 high", "high" in sev(res, "IV04"))
res, _ = check([inv([dict(ok[0], category="books")], {"net": "20.00", "vat": "4.00", "gross": "24.00"})], {"category_rates": {"books": 0}})
t("category rate mismatch -> IV04 medium", "medium" in sev(res, "IV04"))
ten = [{"description": "x%d" % i, "qty": 1, "unit_price": "0.05", "vat_rate": 10} for i in range(10)]
res, _ = check([inv(ten, {"net": "0.50", "vat": "0.05", "gross": "0.55"})], {})
t("per-invoice rounding detected -> IV06 low", "IV06" in codes(res) and "IV05" not in codes(res))
res, _ = check([inv(ten, {"net": "0.50", "vat": "0.10", "gross": "0.60"})], {})
res2, _ = check([inv(ten[:2], {"net": "0.10", "vat": "0.01", "gross": "0.11"})], {})
t("rounding difference within tolerance -> IV06 info", sev(res2, "IV06") == {"info"})
t("per-line rounding: 10 x 0.01 = 0.10 ok", not res[0]["findings"])
res, _ = check([inv(ten, {"net": "0.50", "vat": "0.05", "gross": "0.55"})], {"vat_rounding": "invoice"})
t("vat_rounding=invoice accepts 0.05", not res[0]["findings"])
res, _ = check([inv(ok, {"net": "20.00", "vat": "4.00", "gross": "25.00"})], {})
t("grand total wrong -> IV05 high", "high" in sev(res, "IV05"))
d = [{"description": "a", "qty": 1, "unit_price": "100", "discount_pct": 15, "vat_rate": 20}]
res, _ = check([inv(d, {"net": "85.00", "vat": "17.00", "gross": "102.00"})], {})
t("line discount applied before tax", not res[0]["findings"])
res, _ = check([inv(d, {"net": "85.00", "vat": "17.00", "gross": "102.00"}, invoice_discount_pct="10")], {})
t("invoice discount reduces base", res[0]["expected"]["gross"] == Decimal("91.80"))
res, _ = check([inv(ok + ok, {"net": "40.00", "vat": "8.00", "gross": "48.00"})], {})
t("duplicate line -> IV07 medium", "medium" in sev(res, "IV07"))
a = inv(ok, {"net": "20.00", "vat": "4.00", "gross": "24.00"})
res, _ = check([a, dict(a)], {})
t("duplicate invoice number -> IV07 high", any(f[1] == "IV07" and f[0] == "high" for f in res[1]["findings"]))
res, _ = check([a, dict(a, number="A2")], {})
t("same seller/date/total new number -> IV07 medium", any(f[1] == "IV07" and f[0] == "medium" for f in res[1]["findings"]))
res, _ = check([inv(ok, {"net": "20.00", "vat": "4.00", "gross": "24.00"}, number="", seller={"name": "S"})], {})
t("missing number high, missing vat id medium", "high" in sev(res, "IV01") and "medium" in sev(res, "IV01"))
res, _ = check([inv(ok, {"net": "20.00", "vat": "4.00", "gross": "24.00"}, due_date="2026-09-01")], {}, today=datetime.date(2026, 9, 1))
t("due before date + future date -> IV10", len([f for f in res[0]["findings"] if f[1] == "IV10"]) == 2)
neg = [{"description": "return", "qty": -1, "unit_price": "10", "vat_rate": 20}]
res, _ = check([inv(neg, {"net": "-10.00", "vat": "-2.00", "gross": "-12.00"})], {})
t("negative line on invoice -> IV09", "IV09" in codes(res))
res, _ = check([inv(neg, {"net": "-10.00", "vat": "-2.00", "gross": "-12.00"}, type="credit_note")], {})
t("negative ok on credit note", "IV09" not in codes(res))
g = [{"description": "a", "qty": 1, "unit_price": "12.00", "vat_rate": 20}]
res, _ = check([inv(g, {"net": "10.00", "vat": "2.00", "gross": "12.00"})], {"prices_include_vat": True})
t("tax-inclusive prices", not res[0]["findings"])
t("decimal comma parsing", D("1.234,56") == Decimal("1234.56") and D("12,5") == Decimal("12.5") and D("1,234.56") == Decimal("1234.56"))
csvt = ("invoice_number,invoice_date,seller_name,seller_vat_id,buyer_name,currency,description,qty,unit_price,vat_rate,line_net,vat_amount,total_net,total_vat,total_gross\n"
"C1,2026-10-01,S,X1,B,EUR,a,1,10,20,10.00,2.00,15.00,3.00,18.00\nC1,2026-10-01,S,X1,B,USD,b,1,5,20,5.00,1.00,,,\n")
invs, _, notes = load_invoices(csvt)
res, _ = check(invs, {}, csv_notes=notes)
t("CSV grouped into one invoice", len(invs) == 1 and len(invs[0]["lines"]) == 2 and not any(f[0] == "high" for f in res[0]["findings"]))
t("CSV inconsistent header -> IV13", "IV13" in codes(res))
try:
load_invoices("{not json"); okk = False
except InputError:
okk = True
t("bad JSON rejected", okk)
res, _ = check([inv([dict(ok[0], qty="two")], {"gross": "24"})], {})
t("unparseable qty -> IV01 high", "high" in sev(res, "IV01"))
rep = report(*check([a], {}), "x")
t("report has verdict and not-tax-advice line", "Verdict" in rep and "not tax advice" in rep)
for name, okk in tests:
print(("PASS " if okk else "FAIL ") + name)
n = sum(1 for _, okk in tests if okk)
print("%d/%d passed" % (n, len(tests)))
return 0 if n == len(tests) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("invoices", nargs="?", help="invoices .json or line items .csv")
ap.add_argument("--rules", help="rules JSON (rates, rounding, tolerance, required fields)")
ap.add_argument("--today", help="YYYY-MM-DD for future-date checks (default: today)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.invoices:
ap.error("invoices file required (or --self-test)")
try:
try:
text = open(a.invoices, encoding="utf-8-sig").read()
except OSError as e:
raise InputError(str(e))
invs, emb_rules, notes = load_invoices(text, a.invoices.lower().endswith(".csv") or None)
rules = dict(emb_rules or {})
if a.rules:
try:
rules.update(json.load(open(a.rules, encoding="utf-8")))
except (OSError, ValueError) as e:
raise InputError("rules: %s" % e)
unknown = set(rules) - set(DEFAULT_RULES) - {"name", "note"}
if unknown:
raise InputError("unknown rules key(s): %s" % ", ".join(sorted(unknown)))
if rules.get("vat_rounding", "line") not in ("line", "invoice") or rules.get("rounding", "half_up") not in ("half_up", "half_even"):
raise InputError("vat_rounding must be line|invoice and rounding half_up|half_even")
today = datetime.date.fromisoformat(a.today) if a.today else datetime.date.today()
results, R = check(invs, rules, today, notes)
except (InputError, ValueError) as e:
print("error: %s" % e, file=sys.stderr)
return 2
if a.json:
conv = lambda v: str(v) if isinstance(v, Decimal) else v
out = [{"invoice": r["label"], "seller": r["seller"], "currency": r["currency"],
"stated": {k: conv(v) for k, v in r["stated"].items()}, "recomputed": {k: conv(v) for k, v in r["expected"].items()},
"tax_by_rate": {str(k): [str(v[0]), str(v[1])] for k, v in r["per_rate"].items()},
"findings": [{"severity": s, "code": c, "message": m} for s, c, m in r["findings"]]} for r in results]
print(json.dumps({"invoices": out}, indent=2))
else:
print(report(results, R, a.invoices))
return 1 if any(f[0] == "high" for r in results for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
---
name: conventional-commits-changelog-builder
description: Turns a git log in Conventional Commits format into a ready-to-paste Keep a Changelog release section - Added, Changed, Deprecated, Removed, Fixed, Security - with scopes, issue and commit links and a breaking-changes block. Flags breaking changes without migration notes, non-conforming and fixup commits, reverts and duplicates, and suggests the next SemVer version. Includes an offline stdlib Python builder with --self-test.
---
# Conventional Commits Changelog Builder
You help maintainers turn the commits since the last tag into a changelog entry people can actually read. Writing it by hand means scrolling a log, guessing which commits matter and forgetting the one breaking change hidden in a footer. Generating it blindly copies every typo, WIP commit and reverted feature into the release. You parse the commits, map them to Keep a Changelog sections, list exactly what was left out and why, and say which version the release should get.
## Files in this skill
- `scripts/changelog_build.py` - reads `git log` (default format), `git log --oneline` or a `%x1f`/`%x1e` records export; parses `type(scope)!: description` headers and `BREAKING CHANGE:` footers; groups into Added / Changed / Deprecated / Removed / Fixed / Security with a BREAKING CHANGES block; cancels reverts that are in range, skips merges, drops duplicates, flags fixup/WIP and non-conforming commits with a suggested header; links issues, commits and the compare range; suggests the SemVer bump (0.x aware); can insert the section into an existing CHANGELOG.md (Python 3.8+ standard library only)
- `references/commit-to-changelog-mapping.md` - the type -> section table, breaking-change rules, bump rules and the finding codes CC01-CC10
- `references/release-hygiene.md` - writing entries for users, migration notes, what to do with non-conforming history, and CI checks
- `templates/release-notes-review.md` - the review to hand back with the section
- `examples/git-log.txt` - 21 commits of a fictional CLI (tidyq) since v2.3.1: two breaking changes, a security fix, a deprecation, a revert, a duplicate cherry-pick, a fixup, a WIP and a plain "Fixed ..." commit
- `examples/CHANGELOG.md` - the project's existing changelog (Unreleased + 2.3.1 + 2.3.0)
- `examples/changelog-report.md` - the real builder output on that log
## Workflow
### 1. Get the commit range
Ask for the last released tag and export everything after it, newest first:
```
git log v2.3.1..HEAD > commits.txt
git log v2.3.1..HEAD --format='%H%x1f%s%x1f%b%x1e' > commits.txt # safest: keeps bodies, no parsing guesswork
```
Also ask for the current version, the release date, the repository URL (for links), the tag prefix (`v` by default) and whether internal commits (refactor, docs, ci, chore, test) should be listed. A squash-merge workflow is ideal: one PR = one commit = one line.
### 2. Run the builder
```
python3 scripts/changelog_build.py examples/git-log.txt --current 2.3.1 --date 2026-11-03 --repo-url https://github.com/tidyq/tidyq
python3 scripts/changelog_build.py commits.txt --current 2.3.1 --insert CHANGELOG.md > CHANGELOG.new.md
python3 scripts/changelog_build.py commits.txt --current 0.9.4 --include-internal --json
python3 scripts/changelog_build.py --self-test
```
Exit 0 = ready to tag, 1 = high findings, 2 = bad input. Without a shell, apply `references/commit-to-changelog-mapping.md` by hand: parse each header, map type to section, collect `!` and `BREAKING CHANGE:` footers, cancel reverts, then pick the highest bump.
### 3. Fix the findings before tagging
- CC03 (high): a `!` commit with no `BREAKING CHANGE:` footer gives users no migration path. Write the note (what breaks, what to do instead) into the section.
- CC08 (high): non-conforming commits that look like features or fixes are missing from the changelog. Add them by hand with the suggested header, and turn on a commit-lint check.
- CC07: fixup/WIP commits should be squashed before the release; CC04: reverts of older releases need a sentence explaining why.
- CC01/CC02/CC05/CC06: wording and conformance; fix in the changelog text, not by rewriting published history.
### 4. Edit for humans
The builder gives you correct structure, not good prose. Rewrite entries as user-visible effects ("Export writes Parquet by default" rather than "refactor writer"), merge entries that describe one change, keep scopes that users recognise, and put the BREAKING CHANGES block where nobody can miss it. See `references/release-hygiene.md`.
### 5. Deliver
Fill `templates/release-notes-review.md`: version and why, the final section, the migration notes, what was left out, and follow-ups (commitlint, PR title check). With `--insert`, hand back the updated CHANGELOG.md; the original file is never modified.
## Example result
On `examples/git-log.txt` (current 2.3.1) the builder reads 21 commits, 85% conforming, and suggests **3.0.0**: `feat(export)!` makes Parquet the default (with a proper BREAKING CHANGE footer) and `fix(config)!` drops the INI config file without any migration note (CC03, high). "Fixed crash when the config file is empty" is not a Conventional Commit and would silently miss the changelog (CC08, high; suggested `fix: crash when the config file is empty`). The token-leak fix lands under Security, the `--legacy-dates` deprecation under Deprecated, the colour-themes feature and its revert cancel out, the cherry-picked BOM fix is listed once, and the merge, fixup, WIP and 5 internal commits are listed as left out.
## Rules
- Never invent entries: every line comes from a commit, and every commit is either in the section or in the "left out" list.
- A breaking change always gets a migration note and the major bump (minor while on 0.x); never hide it in Changed.
- Security fixes go under Security even when the commit type is `fix`.
- Do not rewrite published history to fix commit messages; fix the changelog text and add a lint check for the future.
- Keep the newest release on top, dates as YYYY-MM-DD and one compare link per version.
- The version is a suggestion: the maintainer decides, especially for behaviour changes typed as `fix`.
FILE:references/commit-to-changelog-mapping.md
# Commit -> changelog mapping
## Header grammar (Conventional Commits 1.0.0)
```
<type>[(<scope>)][!]: <description>
[body]
[footers] e.g. BREAKING CHANGE: <what breaks and how to migrate>
Closes #123
Refs: #98
```
- `type` is a noun such as feat or fix; lower case by convention.
- Exactly one space after the colon. `fix:missing space` or `fix : x` is malformed (CC01, with a suggested fix).
- `!` before the colon marks a breaking change. A `BREAKING CHANGE:` (or `BREAKING-CHANGE:`) footer also marks one, with or without `!`.
## Type -> Keep a Changelog section
| Commit | Section | Bump |
|---|---|---|
| feat | Added | minor |
| feat with "deprecate" in the description | Deprecated | minor |
| feat / breaking with remove, drop, delete first | Removed | minor / major |
| fix | Fixed | patch |
| fix / deps mentioning security, CVE, vulnerability, XSS, CSRF, injection, leak; type security | Security | patch |
| perf, deps | Changed | patch |
| any type with `!` or BREAKING CHANGE footer | its section + BREAKING CHANGES block | major (minor on 0.x) |
| refactor, docs, style, test, build, ci, chore | not listed (Internal with --include-internal) | none |
| revert of a commit in the same range | both left out | none |
| revert of an older commit | Removed (reverted feat) or Fixed | patch |
| merge commits | left out | none |
| fixup!, squash!, amend!, WIP | left out + CC07 | none |
Aliases accepted with a CC02 warning: feature/features/add -> feat, bugfix/bug/hotfix/fixes/fixed -> fix, doc/documentation -> docs, tests -> test, performance -> perf, refactoring -> refactor, cleanup/release -> chore, dep -> deps. Other unknown types are left out.
## Bump rules
- Highest wins: major > minor > patch. No feat/fix/perf/security/breaking change = no release needed (CC09).
- 1.0.0 and later: breaking -> X+1.0.0. 0.y.z: breaking -> 0.y+1.0 (CC10 reminds you to plan 1.0.0).
- Pre-release and build suffixes on --current are ignored for the bump; add them yourself (`3.0.0-rc.1`).
## Finding codes
| Code | Severity | Meaning |
|---|---|---|
| CC01 | medium | header is not a Conventional Commit (suggested header given) |
| CC02 | medium | unknown type (alias mapped or left out) |
| CC03 | high / info | breaking without BREAKING CHANGE footer (high); footer-only breaking (info) |
| CC04 | info / medium | revert cancelled in range (info); revert of an older release (medium) |
| CC05 | low | duplicate commit (cherry-pick) listed once |
| CC06 | low | style: upper-case type, trailing period, header over 72 chars |
| CC07 | medium | fixup/squash/WIP commit in the release range |
| CC08 | high | non-conforming commits that look like feat/fix are missing from the changelog |
| CC09 | info | no user-facing changes |
| CC10 | info | 0.x major-zero bump rule applied |
FILE:references/release-hygiene.md
# Release hygiene
## Writing entries users understand
- Describe the effect, not the implementation: "Export writes Parquet by default" beats "switch writer backend".
- Start with a verb in the imperative or a noun phrase, capitalise the first word, no trailing period. Keep one change per line.
- Keep scopes users know (cli, export, auth). Replace internal module names with the feature name.
- Merge several commits that describe one change into one entry and keep all links.
- Keep links: issue numbers for context, commit hashes for auditors, one compare link per version at the bottom.
## Migration notes for breaking changes
A good note answers three questions in one or two sentences:
1. What stops working? ("`tidyq export` without --format now writes .parquet")
2. Who is affected? ("scripts that expect a .csv file")
3. What do they do instead? ("pass --format csv")
Put every note in the BREAKING CHANGES block at the top of the release section and link a longer upgrade guide if needed.
## Keep a Changelog basics
- File `CHANGELOG.md`, newest version first, an `## [Unreleased]` section on top for work in progress.
- Version headings `## [3.0.0] - 2026-11-03` (ISO dates), sections in this order: Added, Changed, Deprecated, Removed, Fixed, Security. Leave out empty sections.
- Mark yanked releases `## [2.2.0] - 2026-09-01 [YANKED]`.
## Non-conforming history
- Do not rewrite pushed history to fix messages. Add the missing entries to the changelog by hand (the builder lists them with a suggested header).
- Squash fixup!/WIP commits before merging; with squash-merge, the PR title becomes the commit header, so lint PR titles.
- Cherry-picked commits on release branches appear twice in some ranges; list them once.
## CI checks worth adding
- commitlint (or a PR-title check) with the conventional config on every pull request.
- A release job that runs the builder on `last-tag..HEAD`, fails on CC03/CC08, and opens the changelog diff for review instead of committing it blindly.
- Tag only after the changelog PR is merged, so the tag points at a commit that contains its own notes.
FILE:templates/release-notes-review.md
# Release notes review: <project> <current> -> <next>
**Range:** <last tag>..<commit> **Date:** <YYYY-MM-DD> **Commits:** <n> (<conformance %> conforming)
## Version
<next> - <major/minor/patch> because <breaking change / new feature / fixes only>.
<Any reason to deviate, e.g. behaviour change typed as fix, 0.x rule>
## Breaking changes and migration
| Change | Who is affected | What to do |
|---|---|---|
| <change> | <users/scripts/integrations> | <migration step> |
## Changelog section (final, edited)
```markdown
## [<next>] - <date>
...
```
## Left out on purpose
| Commit | Subject | Why |
|---|---|---|
| <hash> | <subject> | <merge / revert / duplicate / internal / WIP> |
## Fixed before tagging
- [ ] Migration note written for every breaking change (CC03)
- [ ] Missing user-facing commits added by hand (CC08)
- [ ] Fixup/WIP commits squashed (CC07)
## Follow-ups
- <commitlint / PR title check / upgrade guide / announce deprecations>
FILE:examples/git-log.txt
commit 9f1c2a7e4b000000000000000000000000000000
Merge: 8e0b1d6 5b7e8a3
Author: Elif Demir <elif@tidyq.dev>
Date: Mon Nov 2 17:42:10 2026 +0300
Merge pull request #212 from tidyq/feat/parquet-export
commit 8e0b1d6c3a000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Mon Nov 2 16:05:31 2026 +0300
feat(export)!: write Parquet by default for `tidyq export`
Parquet is now the default output format. CSV is still available.
BREAKING CHANGE: `tidyq export` without --format now writes .parquet instead of .csv; pass --format csv to keep the old behaviour.
Refs: #198
commit 7d9a0c5b2f000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Mon Nov 2 11:20:04 2026 +0300
fixup! feat(export)!: write Parquet by default for `tidyq export`
commit 6c8f9b4a1e000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Sun Nov 1 20:14:55 2026 +0300
fix(auth): prevent API token leak in debug logs (#207)
Tokens were printed in full when TIDYQ_DEBUG=1. They are now masked.
Closes #205
commit 5b7e8a3f0d000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Sun Nov 1 15:02:13 2026 +0300
feat(query): support window functions in `--where` (#203)
commit 4a6d7f2e9c000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Sat Oct 31 18:47:39 2026 +0300
perf(reader): stream large CSV files instead of loading them into memory
Peak memory on a 2 GB file drops from 2.6 GB to 180 MB.
Closes #187
commit 3f5c6e1d8b000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Sat Oct 31 12:30:00 2026 +0300
Fixed crash when the config file is empty
commit 2e4b5d0c7a000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Fri Oct 30 19:11:42 2026 +0300
feat(cli): deprecate the --legacy-dates flag
It will be removed in 3.0. Use --date-format instead.
commit 1d3a4c9b6f000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Fri Oct 30 14:25:18 2026 +0300
Revert "feat(ui): colour themes for table output"
This reverts commit 0c2f3b8a5e000000000000000000000000000000.
Themes broke output on Windows terminals.
commit 0c2f3b8a5e000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Fri Oct 30 10:03:51 2026 +0300
feat(ui): colour themes for table output
commit fb1e2a7d4c000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Thu Oct 29 21:40:07 2026 +0300
feature: add `tidyq doctor` command to check the environment
commit ea0d196c3b000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Thu Oct 29 16:12:30 2026 +0300
fix(query): quote column names that contain spaces.
Closes #199
commit d9fc085b2a000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Thu Oct 29 09:55:02 2026 +0300
fix(config)!: drop support for the ~/.tidyqrc INI file
commit c8eb974a19000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Wed Oct 28 18:20:44 2026 +0300
docs: add a Parquet section to the README
commit b7da863f08000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Wed Oct 28 13:08:19 2026 +0300
chore(deps): bump pyarrow from 16.1.0 to 17.0.0
commit a6c9752e97000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Tue Oct 27 17:31:26 2026 +0300
ci: cache pip downloads in the test workflow
commit 95b8641d86000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Tue Oct 27 11:47:53 2026 +0300
test(reader): cover BOM and CRLF inputs
commit 84a7530c75000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Mon Oct 26 15:16:08 2026 +0300
fix(reader): handle UTF-8 BOM at the start of a file (#195)
commit 7396420b64000000000000000000000000000000
Author: Elif Demir <elif@tidyq.dev>
Date: Mon Oct 26 15:16:08 2026 +0300
fix(reader): handle UTF-8 BOM at the start of a file (#195)
commit 62853f9a53000000000000000000000000000000
Author: Sam Okafor <sam@tidyq.dev>
Date: Mon Oct 26 10:02:40 2026 +0300
WIP tweak table widths
commit 5174e8f942000000000000000000000000000000
Author: Mira Chen <mira@tidyq.dev>
Date: Sun Oct 25 19:44:12 2026 +0300
refactor(query): split the parser into tokenizer and planner
FILE:examples/CHANGELOG.md
# Changelog
All notable changes to tidyq are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [2.3.1] - 2026-10-21
### Fixed
- **reader:** Detect semicolon-separated files exported by Excel (#181)
## [2.3.0] - 2026-10-02
### Added
- **query:** `--limit` and `--offset` options (#170)
### Fixed
- **cli:** Exit with code 2 on unknown options instead of 1 (#174)
[2.3.1]: https://github.com/tidyq/tidyq/compare/v2.3.0...v2.3.1
[2.3.0]: https://github.com/tidyq/tidyq/compare/v2.2.0...v2.3.0
FILE:examples/changelog-report.md
# Changelog build: examples/git-log.txt
21 commits read: 17 Conventional (85.0% of non-merge commits), 1 non-conforming, 1 merges, 2 fixup/WIP.
**Version: 2.3.1 -> 3.0.0 (major bump, driven by breaking changes)**
**Verdict: 2 high finding(s) - fix before tagging**
## Release section (paste under the Unreleased heading of CHANGELOG.md)
```markdown
## [3.0.0] - 2026-11-03
### BREAKING CHANGES
- **export:** Write Parquet by default for `tidyq export` - `tidyq export` without --format now writes .parquet instead of .csv; pass --format csv to keep the old behaviour.
- **config:** Drop support for the ~/.tidyqrc INI file - MIGRATION NOTE MISSING
### Added
- **BREAKING** **export:** Write Parquet by default for `tidyq export` ([#198](https://github.com/tidyq/tidyq/issues/198), [8e0b1d6](https://github.com/tidyq/tidyq/commit/8e0b1d6))
- **query:** Support window functions in `--where` ([#203](https://github.com/tidyq/tidyq/issues/203), [5b7e8a3](https://github.com/tidyq/tidyq/commit/5b7e8a3))
- Add `tidyq doctor` command to check the environment ([fb1e2a7](https://github.com/tidyq/tidyq/commit/fb1e2a7))
### Changed
- **reader:** Stream large CSV files instead of loading them into memory ([#187](https://github.com/tidyq/tidyq/issues/187), [4a6d7f2](https://github.com/tidyq/tidyq/commit/4a6d7f2))
### Deprecated
- **cli:** Deprecate the --legacy-dates flag ([2e4b5d0](https://github.com/tidyq/tidyq/commit/2e4b5d0))
### Removed
- **BREAKING** **config:** Drop support for the ~/.tidyqrc INI file ([d9fc085](https://github.com/tidyq/tidyq/commit/d9fc085))
### Fixed
- **query:** Quote column names that contain spaces ([#199](https://github.com/tidyq/tidyq/issues/199), [ea0d196](https://github.com/tidyq/tidyq/commit/ea0d196))
- **reader:** Handle UTF-8 BOM at the start of a file ([#195](https://github.com/tidyq/tidyq/issues/195), [84a7530](https://github.com/tidyq/tidyq/commit/84a7530))
### Security
- **auth:** Prevent API token leak in debug logs ([#205](https://github.com/tidyq/tidyq/issues/205), [#207](https://github.com/tidyq/tidyq/issues/207), [6c8f9b4](https://github.com/tidyq/tidyq/commit/6c8f9b4))
[3.0.0]: https://github.com/tidyq/tidyq/compare/v2.3.1...v3.0.0
```
## Findings
- **HIGH CC03** d9fc085 'fix(config)!: drop support for the ~/.tidyqrc INI file' is breaking (!) but has no 'BREAKING CHANGE:' footer - add a migration note
- **HIGH CC08** 1 non-conforming commit(s) look user-facing (3f5c6e1) and are NOT in the changelog - reword them or add the entries by hand
- **MEDIUM CC01** 3f5c6e1 Fixed crash when the config file is empty: not a Conventional Commit - suggest: fix: crash when the config file is empty
- **MEDIUM CC02** fb1e2a7 feature: add `tidyq doctor` command to check the environment: unknown type 'feature' - treated as 'feat'
- **MEDIUM CC07** 7d9a0c5 'fixup! feat(export)!: write Parquet by default for `tidyq export`' is a fixup/WIP commit on the release branch - squash it before tagging
- **MEDIUM CC07** 62853f9 'WIP tweak table widths' is a fixup/WIP commit on the release branch - squash it before tagging
- **LOW CC05** 7396420 duplicates 84a7530 ('fix(reader): handle UTF-8 BOM at the start of a file (#195)', cherry-pick or re-applied?) - listed once
- **LOW CC06** ea0d196 fix(query): quote column names that contain spaces.: description ends with a period
- **INFO CC04** 1d3a4c9 reverts 0c2f3b8 'feat(ui): colour themes for table output' in the same range - both left out
## Not in the changelog
| Commit | Subject | Why |
|---|---|---|
| 9f1c2a7 | Merge pull request #212 from tidyq/feat/parquet-export | merge commit |
| 7d9a0c5 | fixup! feat(export)!: write Parquet by default for `tidyq export` | fixup/wip commit |
| 3f5c6e1 | Fixed crash when the config file is empty | not a Conventional Commit |
| 1d3a4c9 | Revert "feat(ui): colour themes for table output" | revert (cancelled) |
| 0c2f3b8 | feat(ui): colour themes for table output | reverted |
| 7396420 | fix(reader): handle UTF-8 BOM at the start of a file (#195) | duplicate |
| 62853f9 | WIP tweak table widths | fixup/wip commit |
| - | 5 internal commit(s): chore, ci, docs, refactor, test | internal type (use --include-internal to list) |
FILE:scripts/changelog_build.py
#!/usr/bin/env python3
"""Conventional Commits -> Keep a Changelog release section builder (stdlib only, Python 3.8+).
Reads a commit list (git log default output, git log --oneline, or a records file made with
git log --format='%H%x1f%s%x1f%b%x1e'), parses Conventional Commits headers, groups the changes
into Keep a Changelog sections, flags breaking changes and non-conforming commits, handles
reverts, merges and fixup commits, and suggests the next SemVer version.
Usage:
changelog_build.py COMMITS.txt [--current 1.4.2] [--date 2026-11-03] [--repo-url URL]
[--tag-prefix v] [--include-internal] [--insert CHANGELOG.md] [--json]
changelog_build.py --self-test
Exit: 0 = ok, 1 = high findings (breaking change without migration note, commits that would be
lost from the changelog), 2 = bad input.
"""
import argparse, datetime, json, re, sys
HEADER = re.compile(r"^(?P<type>[A-Za-z][A-Za-z0-9_-]*)(?:\((?P<scope>[^()\s][^()]*)\))?(?P<bang>!)?: (?P<desc>\S.*)$")
LOOSE = re.compile(r"^(?P<type>[A-Za-z][A-Za-z0-9_-]*)\s*(?:\((?P<scope>[^()]*)\))?\s*(?P<bang>!)?\s*:\s*(?P<desc>.*)$")
FOOTER = re.compile(r"^(?P<token>BREAKING[ -]CHANGE|[A-Za-z][A-Za-z-]*)(?:: | #)(?P<value>.+)$")
ISSUE = re.compile(r"(?:(?<=\s)|^|\()#(\d+)\b")
REVERTS = re.compile(r"This reverts commit ([0-9a-f]{7,40})", re.I)
KNOWN = {"feat", "fix", "perf", "refactor", "docs", "style", "test", "build", "ci", "chore", "revert", "security", "deps"}
ALIASES = {"feature": "feat", "features": "feat", "add": "feat", "bugfix": "fix", "bug": "fix", "hotfix": "fix",
"fixes": "fix", "fixed": "fix", "doc": "docs", "documentation": "docs", "tests": "test",
"performance": "perf", "refactoring": "refactor", "cleanup": "chore", "release": "chore", "dep": "deps"}
INTERNAL = {"refactor", "docs", "style", "test", "build", "ci", "chore"}
SECTIONS = ["Added", "Changed", "Deprecated", "Removed", "Fixed", "Security"]
SEV_ORDER = {"high": 0, "medium": 1, "low": 2, "info": 3}
class InputError(Exception):
pass
# ---------------------------------------------------------------- parsing input
def parse_commits(text):
"""Return list of dicts: hash, subject, body (oldest last, i.e. git log order)."""
text = text.replace("\r\n", "\n")
if "\x1e" in text:
out = []
for rec in text.split("\x1e"):
rec = rec.strip("\n")
if not rec.strip():
continue
parts = rec.split("\x1f")
if len(parts) < 2:
raise InputError("record without \\x1f separators: %r" % rec[:60])
out.append({"hash": parts[0].strip(), "subject": parts[1].strip(),
"body": (parts[2] if len(parts) > 2 else "").strip("\n")})
return out
lines = text.split("\n")
if any(re.match(r"^commit [0-9a-f]{7,40}\b", l) for l in lines):
out, cur, msg = [], None, []
def flush():
if cur is not None:
m = [l[4:] if l.startswith(" ") else l.strip() for l in msg]
while m and not m[0].strip():
m.pop(0)
while m and not m[-1].strip():
m.pop()
cur["subject"] = m[0].strip() if m else ""
cur["body"] = "\n".join(m[1:]).strip("\n")
out.append(cur)
for l in lines:
mm = re.match(r"^commit ([0-9a-f]{7,40})\b", l)
if mm:
flush()
cur, msg = {"hash": mm.group(1), "merge": False}, []
elif cur is not None and not msg and re.match(r"^(Merge|Author|AuthorDate|Commit|CommitDate|Date):", l):
if l.startswith("Merge:"):
cur["merge"] = True
if l.startswith("Author:"):
cur["author"] = l[7:].strip()
if l.startswith("Date:"):
cur["date"] = l[5:].strip()
elif cur is not None:
msg.append(l)
flush()
return out
out = []
for l in lines:
if not l.strip() or l.lstrip().startswith("#"):
continue
mm = re.match(r"^\s*(?:\*\s*)?([0-9a-f]{7,40})\s+(.*)$", l)
if mm:
out.append({"hash": mm.group(1), "subject": mm.group(2).strip(), "body": ""})
else:
out.append({"hash": "", "subject": l.strip(), "body": ""})
return out
def parse_footers(body):
"""Footers = trailing paragraph lines 'Token: value' / 'Token #value' / 'BREAKING CHANGE: ...'."""
feet, breaking = [], None
paras = re.split(r"\n\s*\n", body.strip()) if body.strip() else []
if paras:
last = paras[-1].split("\n")
if FOOTER.match(last[0]):
cur = None
for l in last:
m = FOOTER.match(l)
if m:
cur = [m.group("token"), m.group("value").strip()]
feet.append(cur)
elif cur:
cur[1] += " " + l.strip()
for tok, val in feet:
if tok in ("BREAKING CHANGE", "BREAKING-CHANGE"):
breaking = val
if breaking is None: # BREAKING CHANGE paragraph anywhere in the body
m = re.search(r"^BREAKING[ -]CHANGE: (.+?)(?:\n\s*\n|\Z)", body, re.M | re.S)
if m:
breaking = " ".join(m.group(1).split())
return feet, breaking
def classify(c):
"""Annotate one commit dict in place."""
s = c["subject"]
c["issues"] = sorted(set(ISSUE.findall(s + "\n" + c.get("body", ""))), key=int)
c["problems"] = []
if c.get("merge") or re.match(r"^Merge (pull request|branch|remote-tracking branch|tag) ", s):
c["kind"] = "merge"; return c
m = re.match(r'^Revert "(.+)"$', s)
if m:
c["kind"] = "revert"; c["reverted_subject"] = m.group(1)
r = REVERTS.search(c.get("body", "")); c["reverted_hash"] = r.group(1) if r else None
return c
if re.match(r"^(fixup|squash|amend)! ", s) or re.match(r"^(wip|WIP)\b", s):
c["kind"] = "fixup"; return c
m = HEADER.match(s)
feet, breaking = parse_footers(c.get("body", ""))
c["footers"] = feet
if m:
t = m.group("type")
c.update(type=t.lower(), scope=m.group("scope"), bang=bool(m.group("bang")), desc=m.group("desc").strip(),
breaking_note=breaking, kind="cc")
if t != t.lower():
c["problems"].append(("low", "CC06", "type '%s' should be lower case" % t))
if c["type"] not in KNOWN:
alias = ALIASES.get(c["type"])
c["problems"].append(("medium", "CC02", "unknown type '%s'%s" % (t, (" - treated as '%s'" % alias) if alias else " - left out of the changelog")))
if alias:
c["type"] = alias
else:
c["kind"] = "unknown"
if c["type"] == "revert":
c["kind"] = "revert"; c["reverted_subject"] = c["desc"]
r = REVERTS.search(c.get("body", "")); c["reverted_hash"] = r.group(1) if r else None
return c
d = c["desc"]
if d.endswith("."):
c["problems"].append(("low", "CC06", "description ends with a period"))
if len(s) > 72:
c["problems"].append(("low", "CC06", "header is %d chars (keep <= 72)" % len(s)))
c["breaking"] = c["bang"] or breaking is not None
return c
lm = LOOSE.match(s)
c["kind"] = "nonconforming"
if lm and lm.group("desc"):
t = lm.group("type").lower(); t = ALIASES.get(t, t)
fixed = "%s%s%s: %s" % (t, "(%s)" % lm.group("scope").strip() if lm.group("scope") else "", "!" if lm.group("bang") else "", lm.group("desc").strip())
c["suggest"] = fixed
c["problems"].append(("medium", "CC01", "malformed header (spacing/colon) - suggest: %s" % fixed))
else:
c["suggest"] = guess_header(s)
c["problems"].append(("medium", "CC01", "not a Conventional Commit - suggest: %s" % c["suggest"]))
return c
def guess_header(s):
low = s.lower()
if re.match(r"^(fix|fixed|fixes|resolve|resolved|correct)", low) or "bug" in low or "typo" in low:
t = "docs" if re.search(r"readme|docs?\b|typo", low) else "fix"
elif re.match(r"^(add|added|implement|introduce|support|new)", low):
t = "feat"
elif re.match(r"^(update|bump|upgrade)", low) and re.search(r"dep|version|package|lock", low):
t = "chore(deps)"
elif re.match(r"^(remove|delete|drop)", low):
t = "refactor"
elif re.search(r"test", low):
t = "test"
else:
t = "chore"
if t == "fix":
s = re.sub(r"^(fixed|fixes|fix)\s+", "", s, flags=re.I)
d = re.sub(r"^(fixed|fixes|added|adds|updated|updates|removed|removes)\b", lambda m: {"fixed": "fix", "fixes": "fix", "added": "add", "adds": "add", "updated": "update", "updates": "update", "removed": "remove", "removes": "remove"}[m.group(1).lower()], s, flags=re.I)
d = d[0].lower() + d[1:] if d else d
return "%s: %s" % (t, d.rstrip("."))
def section_for(c):
d = c["desc"].lower()
if c["type"] == "security" or (c["type"] in ("fix", "deps") and re.search(r"\b(cve-\d|security|vulnerab|xss|csrf|injection|leak)", d)):
return "Security"
if re.search(r"\bdeprecat", d):
return "Deprecated"
if re.match(r"(remove|drop|delete)\b", d) and (c["type"] == "feat" or c.get("breaking")):
return "Removed"
if c["type"] == "feat":
return "Added"
if c["type"] == "fix":
return "Fixed"
if c["type"] in ("perf", "deps"):
return "Changed"
if c.get("breaking"):
return "Changed"
return None
# ---------------------------------------------------------------- analysis
def bump_version(current, level):
m = re.match(r"^v?(\d+)\.(\d+)\.(\d+)(?:[-+].*)?$", current or "")
if not m:
return None
a, b, c = map(int, m.groups())
if level == "major":
return "%d.0.0" % (a + 1) if a > 0 else "0.%d.0" % (b + 1)
if level == "minor":
return "%d.%d.0" % (a, b + 1)
if level == "patch":
return "%d.%d.%d" % (a, b, c + 1)
return None
def analyze(commits, current=None, include_internal=False):
for c in commits:
classify(c)
findings = []
# reverts: cancel out when the reverted commit is in range
by_hash = {c["hash"][:7]: c for c in commits if c.get("hash")}
by_subject = {}
for c in commits:
by_subject.setdefault(c["subject"], []).append(c)
for c in commits:
if c.get("kind") != "revert":
continue
tgt = None
if c.get("reverted_hash"):
tgt = by_hash.get(c["reverted_hash"][:7])
if tgt is None and c.get("reverted_subject") in by_subject:
tgt = by_subject[c["reverted_subject"]][-1]
if tgt is not None and tgt is not c:
tgt["reverted_by"] = c["hash"][:7] or c["subject"]
c["cancels"] = True
findings.append(("info", "CC04", "%s reverts %s '%s' in the same range - both left out" % (c["hash"][:7] or "?", tgt["hash"][:7] or "?", tgt["subject"])))
else:
c["cancels"] = False
findings.append(("medium", "CC04", "%s reverts '%s' from an earlier release - listed under Removed/Fixed; check the wording" % (c["hash"][:7] or "?", c.get("reverted_subject"))))
seen = {}
for c in commits:
if c.get("kind") == "cc":
key = (c["type"], (c.get("scope") or "").lower(), c["desc"].lower().rstrip("."))
if key in seen:
c["duplicate_of"] = seen[key]
findings.append(("low", "CC05", "%s duplicates %s ('%s', cherry-pick or re-applied?) - listed once" % (c["hash"][:7] or "?", seen[key] or "?", c["subject"])))
else:
seen[key] = c["hash"][:7]
sections = {s: [] for s in SECTIONS}
internal, breaking, skipped = [], [], []
level = None
rank = {"patch": 1, "minor": 2, "major": 3}
def up(l):
nonlocal level
if level is None or rank[l] > rank[level]:
level = l
for c in commits:
k = c.get("kind")
for sev, code, msg in c["problems"]:
findings.append((sev, code, "%s %s: %s" % (c["hash"][:7] or "-", c["subject"], msg)))
if k == "merge":
skipped.append((c, "merge commit")); continue
if k == "fixup":
skipped.append((c, "fixup/wip commit"))
findings.append(("medium", "CC07", "%s '%s' is a fixup/WIP commit on the release branch - squash it before tagging" % (c["hash"][:7] or "-", c["subject"])))
continue
if c.get("reverted_by") or (k == "revert" and c.get("cancels")):
skipped.append((c, "reverted" if c.get("reverted_by") else "revert (cancelled)")); continue
if c.get("duplicate_of"):
skipped.append((c, "duplicate")); continue
if k == "revert":
entry = {"text": "Revert: " + (c.get("reverted_subject") or c["subject"]), "hash": c["hash"][:7], "issues": c["issues"], "scope": None}
sections["Removed" if (c.get("reverted_subject") or "").startswith("feat") else "Fixed"].append(entry)
up("patch"); continue
if k in ("nonconforming", "unknown"):
skipped.append((c, "not a Conventional Commit")); continue
sec = section_for(c)
entry = {"text": c["desc"], "hash": c["hash"][:7], "issues": c["issues"], "scope": c.get("scope"),
"breaking": c.get("breaking", False), "type": c["type"]}
if c.get("breaking"):
up("major")
breaking.append({"hash": c["hash"][:7], "scope": c.get("scope"), "desc": c["desc"], "note": c.get("breaking_note")})
if not c.get("breaking_note"):
findings.append(("high", "CC03", "%s '%s' is breaking (!) but has no 'BREAKING CHANGE:' footer - add a migration note" % (c["hash"][:7] or "-", c["subject"])))
elif not c["bang"]:
findings.append(("info", "CC03", "%s '%s' is breaking via footer only - consider '!' in the header so it shows in one-line logs" % (c["hash"][:7] or "-", c["subject"])))
if sec:
sections[sec].append(entry)
if c["type"] == "feat":
up("minor")
elif c["type"] in ("fix", "perf", "security", "deps"):
up("patch")
else:
internal.append(entry)
if include_internal:
up("patch")
lost = [c for c, why in skipped if why == "not a Conventional Commit"]
lost_userfacing = [c for c in lost if re.match(r"^(feat|fix|perf|security)", c.get("suggest") or "")]
if lost_userfacing:
findings.append(("high", "CC08", "%d non-conforming commit(s) look user-facing (%s) and are NOT in the changelog - reword them or add the entries by hand" % (len(lost_userfacing), ", ".join(c["hash"][:7] or c["subject"][:20] for c in lost_userfacing))))
if not any(sections.values()) and not breaking:
findings.append(("info", "CC09", "no user-facing changes (only internal commits) - a release is optional"))
nxt = bump_version(current, level) if current else None
if current and level == "major" and re.match(r"^v?0\.", current):
findings.append(("info", "CC10", "version is 0.x: breaking changes bump the minor (%s); go to 1.0.0 once the API is stable" % nxt))
if current and not re.match(r"^v?\d+\.\d+\.\d+", current):
raise InputError("--current must look like 1.4.2, got %r" % current)
findings.sort(key=lambda f: (SEV_ORDER[f[0]], f[1]))
stats = {"commits": len(commits),
"conventional": sum(1 for c in commits if c.get("kind") in ("cc", "revert")),
"nonconforming": sum(1 for c in commits if c.get("kind") in ("nonconforming", "unknown")),
"merges": sum(1 for c in commits if c.get("kind") == "merge"),
"fixups": sum(1 for c in commits if c.get("kind") == "fixup")}
checkable = len(commits) - stats["merges"]
stats["conformance_pct"] = round(100.0 * stats["conventional"] / checkable, 1) if checkable else 100.0
return {"sections": sections, "internal": internal, "breaking": breaking, "skipped": skipped,
"level": level, "current": current, "next": nxt, "findings": findings, "stats": stats, "commits": commits}
# ---------------------------------------------------------------- rendering
def clean_text(t):
"""Drop a trailing '(#123)' (it is linked separately) and a trailing period."""
t = re.sub(r"\s*\((?:#\d+(?:,\s*)?)+\)\s*$", "", t).rstrip()
return t[:-1] if t.endswith(".") and not t.endswith("..") else t
def fmt_entry(e, repo_url=None):
t = clean_text(e["text"])
t = t[0].upper() + t[1:] if t else t
s = ("**%s:** " % e["scope"] if e.get("scope") else "") + t
if e.get("breaking"):
s = "**BREAKING** " + s
refs = []
for i in e.get("issues") or []:
refs.append("[#%s](%s/issues/%s)" % (i, repo_url, i) if repo_url else "#%s" % i)
if e.get("hash"):
refs.append("[%s](%s/commit/%s)" % (e["hash"], repo_url, e["hash"]) if repo_url else e["hash"])
return "- " + s + (" (" + ", ".join(refs) + ")" if refs else "")
def release_section(r, date=None, repo_url=None, tag_prefix="v", include_internal=False):
ver = r["next"] or "Unreleased"
head = "## [%s] - %s" % (ver, date or datetime.date.today().isoformat()) if r["next"] else "## [Unreleased]"
out = [head, ""]
if r["breaking"]:
out += ["### BREAKING CHANGES", ""]
for b in r["breaking"]:
d = clean_text(b["desc"])
line = "- " + ("**%s:** " % b["scope"] if b["scope"] else "") + d[0].upper() + d[1:]
line += " - " + b["note"] if b["note"] else " - MIGRATION NOTE MISSING"
out.append(line)
out.append("")
for s in SECTIONS:
items = sorted(r["sections"][s], key=lambda e: (not e.get("breaking"), (e.get("scope") or "~").lower()))
if items:
out += ["### " + s, ""] + [fmt_entry(e, repo_url) for e in items] + [""]
if include_internal and r["internal"]:
out += ["### Internal", ""] + [fmt_entry(e, repo_url) for e in r["internal"]] + [""]
if r["next"] and repo_url and r["current"]:
out.append("[%s]: %s/compare/%s%s...%s%s" % (ver, repo_url, tag_prefix, r["current"].lstrip("v"), tag_prefix, ver))
out.append("")
return "\n".join(out).rstrip() + "\n"
def insert_into(changelog_text, section, version):
if version and re.search(r"^## \[%s\]" % re.escape(version), changelog_text, re.M):
raise InputError("CHANGELOG already has a section for %s" % version)
lines = changelog_text.split("\n")
for i, l in enumerate(lines):
if re.match(r"^## \[Unreleased\]", l, re.I):
j = i + 1
while j < len(lines) and not lines[j].startswith("## "):
j += 1
leftover = [x for x in lines[i + 1:j] if x.strip()]
new = lines[:i + 1] + [""] + section.rstrip("\n").split("\n") + [""] + lines[j:]
return "\n".join(new), leftover
for i, l in enumerate(lines):
if l.startswith("## "):
return "\n".join(lines[:i] + section.rstrip("\n").split("\n") + [""] + lines[i:]), []
return changelog_text.rstrip("\n") + "\n\n" + section, []
def report(r, args):
st = r["stats"]
high = [f for f in r["findings"] if f[0] == "high"]
out = ["# Changelog build: %s" % args.commits, ""]
out.append("%d commits read: %d Conventional (%.1f%% of non-merge commits), %d non-conforming, %d merges, %d fixup/WIP." % (
st["commits"], st["conventional"], st["conformance_pct"], st["nonconforming"], st["merges"], st["fixups"]))
out.append("")
if r["current"]:
out.append("**Version: %s -> %s (%s bump%s)**" % (r["current"], r["next"] or r["current"], r["level"] or "no",
", driven by " + {"major": "breaking changes", "minor": "new features", "patch": "fixes/perf"}[r["level"]] if r["level"] else ""))
else:
out.append("**Suggested bump: %s** (pass --current X.Y.Z for the version number)" % (r["level"] or "none"))
out.append("")
out.append("**Verdict: %s**" % ("%d high finding(s) - fix before tagging" % len(high) if high else "ready to tag"))
out += ["", "## Release section (paste under the Unreleased heading of CHANGELOG.md)", "", "```markdown",
release_section(r, args.date, args.repo_url, args.tag_prefix, args.include_internal).rstrip(), "```", ""]
out += ["## Findings", ""]
if r["findings"]:
out += ["- **%s %s** %s" % (s.upper(), c, m) for s, c, m in r["findings"]]
else:
out.append("- none")
out += ["", "## Not in the changelog", "", "| Commit | Subject | Why |", "|---|---|---|"]
for c, why in r["skipped"]:
out.append("| %s | %s | %s |" % (c["hash"][:7] or "-", c["subject"].replace("|", "\\|"), why))
if r["internal"] and not args.include_internal:
out.append("| - | %d internal commit(s): %s | internal type (use --include-internal to list) |" % (
len(r["internal"]), ", ".join(sorted({e["type"] for e in r["internal"]}))))
out.append("")
return "\n".join(out)
# ---------------------------------------------------------------- self-test
def self_test():
tests = []
def t(name, cond):
tests.append((name, bool(cond)))
def run(txt, cur="1.4.2"):
return analyze(parse_commits(txt), cur)
r = run("a1b2c3d feat(api): add export endpoint\nb2c3d4e fix: handle empty list\n")
t("oneline parse + feat -> minor", r["next"] == "1.5.0" and len(r["sections"]["Added"]) == 1)
t("fix -> Fixed section", r["sections"]["Fixed"][0]["text"] == "handle empty list")
t("entry text drops trailing (#12) and period", fmt_entry({"text": "quote names. (#12)", "issues": ["12"], "hash": "abc1234"}) == "- Quote names (#12, abc1234)")
r = run("a1b2c3d fix: x\n")
t("fix only -> patch", r["next"] == "1.4.3")
r = run("a1b2c3d feat!: drop node 16\n")
t("bang breaking -> major", r["next"] == "2.0.0" and r["breaking"])
t("bang without footer is high CC03", any(f[1] == "CC03" and f[0] == "high" for f in r["findings"]))
t("feat removing -> Removed", r["sections"]["Removed"])
rec = "abc1234\x1ffeat(cli): new flag\x1fBody text.\n\nBREAKING CHANGE: --old is gone, use --new\x1e"
r = run(rec)
t("records format + footer breaking", r["breaking"] and r["breaking"][0]["note"].startswith("--old"))
t("footer-only breaking is info not high", not any(f[0] == "high" for f in r["findings"]))
t("0.x breaking -> minor", bump_version("0.9.3", "major") == "0.10.0")
r = run("a1b2c3d Fixed crash when saving\nb2c3d4e chore: tidy\n")
t("non-conforming flagged CC01", any(f[1] == "CC01" for f in r["findings"]))
t("suggest drops 'Fixed' verb", any("suggest: fix: crash when saving" in f[2] for f in r["findings"]))
t("user-facing non-conforming is high CC08", any(f[1] == "CC08" and f[0] == "high" for f in r["findings"]))
r = run("a1b2c3d feature: dark mode\n")
t("alias feature -> feat", r["sections"]["Added"] and any(f[1] == "CC02" for f in r["findings"]))
r = run("a1b2c3d fix:missing space\n")
t("malformed spacing suggestion", any("suggest: fix: missing space" in f[2] for f in r["findings"]))
log = ("commit 1111111aaaa\nAuthor: A <a@x>\nDate: Mon\n\n Revert \"feat: shiny\"\n\n This reverts commit 2222222bbbb.\n\n"
"commit 2222222bbbb\nAuthor: A <a@x>\nDate: Mon\n\n feat: shiny\n")
r = run(log)
t("revert in range cancels both", not any(r["sections"].values()) and any(f[1] == "CC04" for f in r["findings"]))
r = run("commit 3333333\nMerge: 1 2\nAuthor: A\nDate: x\n\n Merge pull request #5 from a/b\n\ncommit 4444444\nAuthor: A\nDate: x\n\n fixup! fix: y\n")
t("merge skipped, fixup flagged CC07", r["stats"]["merges"] == 1 and any(f[1] == "CC07" for f in r["findings"]))
r = run("a1b2c3d fix(auth): prevent token leak in logs (#41)\n")
t("security keyword -> Security + issue ref", r["sections"]["Security"] and r["sections"]["Security"][0]["issues"] == ["41"])
r = run("a1b2c3d feat(cfg): deprecate legacy yaml keys\n")
t("deprecate -> Deprecated", r["sections"]["Deprecated"])
r = run("a1b2c3d docs: readme\nb2c3d4e ci: cache\n")
t("internal only -> no bump + CC09", r["level"] is None and any(f[1] == "CC09" for f in r["findings"]))
r = run("a1b2c3d fix: same\nb2c3d4e fix: same\n")
t("duplicate listed once", len(r["sections"]["Fixed"]) == 1)
sec = release_section(run("a1b2c3d feat: x\n"), "2026-11-03", "https://github.com/o/r")
t("compare link + date", "## [1.5.0] - 2026-11-03" in sec and "compare/v1.4.2...v1.5.0" in sec)
new, left = insert_into("# Changelog\n\n## [Unreleased]\n\n## [1.4.2] - 2026-10-01\n- a\n", sec, "1.5.0")
t("insert after Unreleased", new.index("## [Unreleased]") < new.index("## [1.5.0]") < new.index("## [1.4.2]"))
try:
insert_into("## [1.5.0] - x\n", sec, "1.5.0"); ok = False
except InputError:
ok = True
t("insert refuses existing version", ok)
try:
run("a fix: x\n", cur="one"); ok = False
except InputError:
ok = True
t("bad --current rejected", ok)
for name, okk in tests:
print(("PASS " if okk else "FAIL ") + name)
n = sum(1 for _, okk in tests if okk)
print("%d/%d passed" % (n, len(tests)))
return 0 if n == len(tests) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("commits", nargs="?")
ap.add_argument("--current", help="current released version, e.g. 1.4.2")
ap.add_argument("--date", help="release date (default today)")
ap.add_argument("--repo-url", help="https://github.com/owner/repo for commit, issue and compare links")
ap.add_argument("--tag-prefix", default="v")
ap.add_argument("--include-internal", action="store_true", help="also list refactor/docs/ci/chore/test commits")
ap.add_argument("--insert", metavar="CHANGELOG.md", help="print the changelog with the new section inserted (file is not modified)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.commits:
ap.error("commits file required (or --self-test)")
try:
if a.date and not re.match(r"^\d{4}-\d{2}-\d{2}$", a.date):
raise InputError("--date must be YYYY-MM-DD")
try:
text = sys.stdin.read() if a.commits == "-" else open(a.commits, encoding="utf-8").read()
except OSError as e:
raise InputError(str(e))
commits = parse_commits(text)
if not commits:
raise InputError("no commits found in %s" % a.commits)
r = analyze(commits, a.current, a.include_internal)
if a.insert:
try:
old = open(a.insert, encoding="utf-8").read()
except OSError as e:
raise InputError(str(e))
new, leftover = insert_into(old, release_section(r, a.date, a.repo_url, a.tag_prefix, a.include_internal), r["next"])
if leftover:
sys.stderr.write("note: %d hand-written line(s) under [Unreleased] were kept below the new section - merge them in\n" % len(leftover))
sys.stdout.write(new if new.endswith("\n") else new + "\n")
elif a.json:
j = {k: r[k] for k in ("sections", "internal", "breaking", "level", "current", "next", "stats")}
j["findings"] = [{"severity": s, "code": c, "message": m} for s, c, m in r["findings"]]
j["skipped"] = [{"hash": c["hash"][:7], "subject": c["subject"], "why": w} for c, w in r["skipped"]]
j["release_section"] = release_section(r, a.date, a.repo_url, a.tag_prefix, a.include_internal)
print(json.dumps(j, indent=2))
else:
print(report(r, a))
except InputError as e:
print("error: %s" % e, file=sys.stderr)
return 2
return 1 if any(f[0] == "high" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
Start from the step 2 image "Whitewashed Aegean Cliffside Village with a Blue-Domed Chapel at Sunset" as the exact first frame, and animate it into a calm 6-second cinematic shot. Keep the white cube houses with blue doors on the left, the blue-domed chapel at the top-left, the winding stone stairway with magenta bougainvillea, the dark cliff, the calm deep blue sea, the sun just above the horizon, the colors, and the style exactly as in the image. The camera makes a slow, smooth push-in and gentle drift toward the village and the stone stairway, ending close on a lit doorway with warm wall lanterns and bougainvillea beside the steps. The small white ferry glides slowly to the right across the sea, leaving its thin wake. Sunlight glitters on the water, the clouds drift slightly, and more windows and lamps begin to glow warm in the village. The shot ends warm, peaceful, and romantic in the golden evening light. Natural, realistic motion, no people, no cuts, no text. Soft sound of gentle waves, a light sea breeze, and distant evening calm.Source: github.com/f/prompts.chat · Contributor: f
A whitewashed Aegean cliffside village at sunset, with white cube houses and blue doors cascading down a steep volcanic cliff, a blue-domed chapel, a winding stone stairway with magenta bougainvillea, and a calm deep blue sea with a small white ferry, wide landscape 16:9. The village fills the left side of the frame, stepping down the cliff in terraces with flat roofs, low white walls, and small courtyards. A small white chapel with a bright blue dome and a cross stands at the top-left. A winding stone stairway with low white walls leads down through the village in the lower left-center, lined with big clusters of magenta bougainvillea. A few windows, doorways, and wall lamps glow warm. The dark rocky cliff drops into a calm deep blue sea that fills the right side, where one small white ferry sails mid-right, leaving a thin wake. The sun sits just above the horizon on the right, casting a golden path on the water under an orange, pink, and violet sky with long thin clouds, and hazy islands and headlands line the horizon. Photoreal golden hour travel photography, warm, peaceful, romantic mood. No people, no text, no logos.
Source: github.com/f/prompts.chat · Contributor: f
Act as a cinematic travel scene designer who writes image-ready and motion-ready scene briefs. Help me design one hero shot of a whitewashed Aegean cliffside village at sunset: white cube houses with blue doors cascading down a steep volcanic cliff, a blue-domed chapel at the top, a winding stone stairway with magenta bougainvillea, and a calm deep blue sea with a small ferry under an orange and pink sky. The shot will first be generated as a still image and then animated into a short video that uses that image as its first frame. Your brief must make the still image rich in elements that can move naturally (the ferry gliding and leaving a wake, sunlight glittering on the sea, lamps and windows lighting up, soft clouds drifting, bougainvillea stirring in a light breeze, a slow camera move toward the stairway) while keeping the composition calm, balanced, readable, and easy to animate. My starting idea: a quiet summer evening in a whitewashed village on a Greek island cliff, with the sun touching the horizon and a small ferry crossing the sea below My starting choices: - Village: white cube houses with flat roofs, blue doors, and small terraces stacked down a steep cliff on the left side of the frame - Chapel: a small white chapel with a bright blue dome and a cross at the top-left - Stairway: a winding stone stairway with low white walls leading down through the village in the lower left-center - Flowers: big clusters of magenta bougainvillea along the walls and the stairway - Sea and boat: a calm deep blue sea filling the right side, with one small white ferry mid-right leaving a thin wake - Distance: dark volcanic cliffs curving away and hazy islands and headlands on the horizon - Sky and sun: the sun just above the horizon on the right, an orange, pink, and violet sky with long thin clouds - Lights: a few windows and wall lanterns already glowing warm in the village - Visual style: photoreal travel photography, golden hour, natural but rich colors - Mood words: warm, peaceful, romantic, timeless, summer evening First ask me, in one short message, for anything missing: the camera height and position (on a terrace, on the stairway, or above the village), how much of the frame the village and the sea should take, how many boats I want and where, how low the sun should be, how many lights should be on, whether the camera should push in toward the stairway or stay still in the video, where the camera move should end (for example a lit doorway with lanterns), and anything to avoid (people, faces, text, signs with words, logos, brand names, cars, crowds, or copying a specific real hotel, photograph, or named place). Then produce the brief with these sections: 1. **Scene Overview**: two or three sentences on the place, the moment, the light, and the feeling. Describe an imagined village in the spirit of the Greek Cycladic islands, not a copy of a specific real village, hotel, or famous photograph. 2. **The Village and Chapel**: the white cube houses, blue doors and window frames, terraces, how the buildings step down the cliff, and the blue-domed chapel at the top-left. 3. **Stairway, Walls, and Bougainvillea**: the winding stone steps, the low white walls, the lit doorways and wall lanterns, and where the magenta bougainvillea grows. 4. **Cliff, Sea, and Ferry**: the dark volcanic cliff face, the calm deep blue sea, the small white ferry and its wake, and the distant islands and headlands. 5. **Sky and Sun**: the position of the sun above the horizon, the colors and shapes of the clouds, and the path of sunlight on the water. 6. **Light and Color**: warm sunset light on white walls, cool blue shadows, glowing windows and lanterns, and the overall exposure of golden hour without blown-out whites. 7. **Fixed Visual Details**: a numbered list of 8 to 10 details that must stay identical in the still image and in the video (camera position, village on the left, chapel top-left, stairway lower left-center, bougainvillea clusters, sea on the right, ferry mid-right, sun position, horizon line, framing). 8. **Motion Map**: a table of every element that can move in the video, with columns: element, type of motion, direction, speed (very slow, slow, medium), and how strongly it should move. Include the ferry gliding to the right with its wake, sunlight glittering on the sea, lamps and windows glowing brighter, clouds drifting, bougainvillea stirring, and the camera move. Keep total motion subtle and physically plausible. 9. **Prompt Seed A - Hero Still Image**: a ready-to-paste image prompt under 170 words for a wide landscape 16:9 frame, with the whitewashed cliffside village, the blue-domed chapel, the stone stairway with bougainvillea, and the sea with the ferry at sunset in the first sentence, ending with: no people, no text, no logos. 10. **Prompt Seed B - Image-to-Video Animation**: a ready-to-paste video prompt under 120 words that starts from the hero still as the first frame, describes a slow, smooth camera push-in for about 6 seconds toward the village and the stone stairway that ends close on a lit doorway with wall lanterns and bougainvillea, the ferry gliding slowly to the right, sunlight glittering on the sea, more lamps glowing warm, and says to keep the houses, chapel, stairway, cliff, and colors exactly as in the image with no people. Rules: no people, faces, text, signs with words, or logos; no cars, crowds, or modern brand objects; no fast camera moves or cuts; and no motion that breaks physics (the houses, chapel, stairway, and cliff never move or change shape, the ferry moves forward smoothly in one direction and its wake trails behind it, the sun stays near the horizon, lamps glow steadily and do not flicker like fire, flowers stir only gently). Use short paragraphs and lists.
Source: github.com/f/prompts.chat · Contributor: f
A charcoal drawing of a jazz trio playing in a smoky club, landscape 4:3, on warm off-white textured drawing paper. On the left, a pianist in a dark jacket sits on a bench at a grand piano with its lid raised, bent over the keys. In the center, an upright bass player stands with his double bass, plucking the strings, lit from above by a single hanging pendant lamp with a dark shade that casts a soft cone of light. On the right, a drummer sits behind a small drum kit with a bass drum, snare, toms, and two cymbals on stands, sticks in motion. Thick swirling smoke and haze fill the dark background, smudged and blended with the side of the charcoal. Loose, gestural, scribbled lines, cross-hatching on the floor, deep blacks against the glowing center, visible paper grain, moody and intimate late-night atmosphere. Faces in soft shadow with no detailed features, no text, no logos.
Source: github.com/f/prompts.chat · Contributor: f
A photorealistic photograph of a single beam of sunlight falling into a narrow sandstone slot canyon, tall portrait 3:4. A bright shaft of light pours down from a small crack in the rock at the top center and lands as a glowing round spot on the soft sandy floor near the bottom. Fine dust and sand particles swirl and sparkle inside the beam, making it look solid and misty. The canyon walls rise steeply on both sides in smooth, flowing, wave-like curves and layered stripes, glowing deep orange and red where the light bounces and fading to cool violet and plum in the shadows. The narrow passage curves away behind the beam into darkness. Low, wide-angle view from the canyon floor, long exposure, rich warm colors, high dynamic range, fine detail in the rock texture, quiet, sacred, awe-filled mood. No people, no text, no logos.
Source: github.com/f/prompts.chat · Contributor: f
A hand-painted Delft blue tile panel showing a three-masted sailing ship on choppy waves beside a shore of tulip fields and a small Dutch village, wide landscape 16:9. The panel is a grid of 6 by 4 square glazed tiles with thin grout lines, each tile edge decorated with small cobalt corner ornaments, and larger curling flourishes in the four outer corners. On the left, a tall wooden three-masted ship with billowing square sails, rigging lines, and small pennants on the mastheads sails through rolling, white-capped waves that fill the whole bottom row of tiles. On the right, a flat shoreline stretches across the middle of the panel with long, straight rows of tulip fields leading toward a village of gabled houses and trees with a single slender church spire. Puffy, softly shaded clouds float across a pale sky, with a few seagulls gliding between them. Painted only in cobalt blue on milky white glaze, with soft brush washes, fine line detail, and a delicate crackle in the glaze, traditional Dutch pottery look. No text, no lettering, no logos.
Source: github.com/f/prompts.chat · Contributor: f
role: Friendly, well-organized Friendsgiving and potluck dinner coordinator who helps hosts plan a shared dinner with friends, covering the guest list, a balanced dish sign-up, dietary needs and allergens, oven and fridge space, equipment, a day-of timeline, leftovers, and cleanup roles, with a calm plan that keeps the host relaxed and makes every guest feel included task: > Turn my guest list, my home and kitchen setup, the dishes people offered to bring, everyone's dietary needs, my budget, and my date and time into a complete potluck dinner coordination plan as YAML: a summary with the main risks, a guest list with what each person brings, a balanced dish sign-up, a dietary and allergen matrix, an oven, stove, and fridge schedule, an equipment and seating list, a shopping list for the host, a day-of timeline, a leftovers and containers plan, cleanup roles, ready-to-send messages for guests, and a final checklist. inputs: event_and_date: Friendsgiving dinner at my apartment on Saturday, November 21, guests arrive at 17:00, dinner at 18:30 guests: 12 adults, including two couples, one guest bringing a 7-year-old, and two friends who are new to the group dishes_offered: 3 people want to bring desserts, 2 want to bring mac and cheese, one will bring wine, the rest have not said host_dishes: I will roast a turkey and make gravy dietary_needs: one vegan, one gluten-free, one guest with a severe tree nut allergy, one guest who does not drink alcohol kitchen_setup: one standard oven, four stove burners, a microwave, and a small fridge that is already half full space_and_seating: a dining table for 6, a sofa, and a coffee table, no balcony equipment_i_have: 8 plates, 6 chairs, one large roasting pan, a slow cooker, not enough serving spoons budget: about 120 for the turkey, drinks, ice, and supplies travel_and_timing_limits: two guests arrive late from work at about 18:00, some guests come by public transport and cannot carry hot dishes far host_stress_points: I worry about everything being ready at the same time and the kitchen getting crowded ask_first: - The date, start and dinner times, and how many adults and children are coming, including anyone who may arrive late or leave early - What each guest has offered to bring, whether it will arrive hot, cold, or ready to serve, and whether it needs oven, stove, microwave, or fridge time - Every dietary need and allergy, how severe each allergy is, and whether anyone avoids alcohol or certain foods for religious or personal reasons - Your kitchen, fridge, freezer, seating, and table space, and the plates, cutlery, glasses, serving dishes, and containers you already have - Your budget, what you will cook yourself, and the jobs you would like guests to help with before, during, and after dinner rules: - Use only my inputs for facts. If something is missing, write it under assumptions and add a question to open_questions. Never invent guest names, preferences, or recipes I did not give. - Balance the dish sign-up across mains, sides, vegetables, salads, bread, desserts, drinks, and ice, with enough food for every guest plus a small extra. Flag duplicates and gaps kindly and suggest what to swap. - Make sure every guest with a dietary need has at least one main or filling dish and one dessert they can eat, marked clearly. - For allergies, ask every cook to list all ingredients, label every dish on the table, keep allergen-free dishes and serving spoons separate, and suggest the guest with a severe allergy checks dishes and carries any medication they need. Never say a dish is safe; say it is labeled and to check with the cook. - Build the oven, stove, microwave, and fridge schedule in 15 or 30 minute slots so no two dishes need the same space at the same time, and prefer dishes that arrive ready to serve, cold, or in a slow cooker when space is tight. - Keep food safety simple, such as keeping hot food hot and cold food cold, not leaving perishable food out for more than about two hours, and cooling leftovers quickly, and tell me to follow official food safety advice where I live. - The timeline must start the week before, include the morning and afternoon of the event, and end after cleanup, with the host's own cooking steps and the moments guests arrive. - Share cleanup roles fairly, never give all of them to the host, and keep jobs simple for children and new guests. - Messages must be short, warm, and clear, for example the sign-up invite, a reminder with what to bring, and a thank-you note. - Output only valid YAML matching output_format. No prose outside the YAML. output_format: summary: event: string date: string guests_count: number dinner_time: string main_risks: - string readiness: on_track | needs_sign_ups | over_capacity assumptions: - string open_questions: - string guests: - name: string arrives_at: string bringing: string dietary_needs: - string allergies: - string helper_role: string dish_sign_up: - course: main | side | vegetable | salad | bread | dessert | drink | ice dish: string cook: string serves: number arrives: hot | cold | room_temperature | needs_finishing gap_or_duplicate: string dietary_and_allergen_matrix: - dish: string vegan: boolean vegetarian: boolean gluten_free: boolean contains_nuts: boolean contains_dairy: boolean contains_alcohol: boolean other_allergens: - string label_ready: boolean kitchen_schedule: oven: - slot: string dish: string temperature: string owner: string stove_and_microwave: - slot: string dish: string appliance: string owner: string fridge_and_cold_storage: - item: string space_needed: string from: string until: string backup_option: string equipment_and_seating: have: - string borrow_or_ask_guests: - item: string from_whom: string buy: - item: string quantity: string estimated_cost: number seating_plan: - string host_shopping_list: - item: string quantity: string estimated_cost: number buy_by: string day_of_timeline: - time: string task: string owner: string leftovers_plan: containers_needed: number cooling_steps: - string share_out_rules: - string use_by_notes: - string cleanup_roles: - role: string person: string when: string messages: sign_up_invite: string reminder: string allergy_note_to_cooks: string thank_you: string final_checklist: one_week_before: - string two_days_before: - string morning_of: - string one_hour_before: - string after_dinner: - string
Source: github.com/f/prompts.chat · Contributor: f
{
"role": "Calm, detail-focused rental car inspection and dispute assistant who helps travelers check a rental car at pickup and return, record clear photo and video evidence, check fuel and mileage, review damage claims and deposit holds, and write polite, firm dispute letters, without inventing rental company policies, fees, laws, or insurance terms",
"task": "Turn my rental booking details, what I see at pickup and return, my evidence, and any damage claim or deposit hold into a single strict JSON inspection and dispute report: a summary, a pickup inspection checklist, a photo and video evidence plan, fuel and mileage checks, a return inspection checklist, a damage claim response, a deposit hold check, insurance and coverage questions to confirm, ready-to-send letters, a timeline of deadlines, and a next steps list. Never invent rental company policies, fees, repair costs, insurance coverage, or laws.",
"context": {
"rental_company_and_location": "a major rental company at an airport location",
"booking_details": "compact car, 5 days, pickup Monday 10:00, return Saturday 18:00 at the same airport",
"stage": "pickup | return | after_return_claim",
"pickup_condition_notes": "a small scratch on the rear bumper and a chip in the windshield, both marked on the pickup form",
"return_condition_notes": "returned after hours using the key drop box, the car was clean and undamaged as far as I could see",
"fuel_and_mileage": "full-to-full fuel policy, pickup mileage 23,410 km, unlimited mileage included",
"evidence_available": "a pickup video walk-around with timestamps, 14 photos at return, a fuel receipt from a station 3 km from the airport",
"damage_claim": "two weeks later the company emailed a claim of 480 for a dent on the front passenger door, with one photo and no repair invoice",
"deposit_hold": "a 300 hold was placed on my credit card at pickup and has not been released after 10 days",
"coverage": "I declined the counter insurance, my credit card may include rental car coverage, I have not checked yet",
"payment_method": "credit card",
"country_or_region": "Spain"
},
"ask_first": [
"Which stage you are at (pickup, return, or a claim after return), the rental company, location, and dates and times of pickup and return",
"Every mark or problem you noticed at pickup and whether staff recorded it on the rental agreement or damage form",
"What evidence you have: photos, videos, timestamps, the signed rental agreement, the pickup and return forms, fuel receipts, and any messages with the company",
"The full damage claim or extra charge you received: the amount, the description, the photos, any repair estimate or invoice, and the deadline to reply",
"The deposit amount, when it was placed, how you paid, and which insurance or coverage you bought, declined, or may have through a credit card or travel insurance"
],
"rules": [
"Use only my inputs for facts. If something is missing, set it to null, add it to open_questions, and never guess company policies, fees, repair costs, insurance terms, deposit release times, or laws.",
"Pickup checklist: cover the body panels, bumpers, wheels and tires, windshield and windows, mirrors, lights, interior, seats, trunk, spare tire and tools, dashboard warning lights, fuel level, and mileage. Tell me to have every existing mark recorded on the rental form and signed by staff before driving away.",
"Evidence plan: a slow, continuous video walk-around with the date and time visible or stated aloud, close-up photos of every mark with a reference object for size, wide photos showing each side of the car and the location, photos of the fuel gauge and odometer, and a backup of all files the same day. Repeat at return, especially for after-hours drop-offs.",
"Fuel and mileage: compare the pickup and return levels with the fuel policy and mileage limit I gave, keep fuel receipts with time and location, and flag any refuel charge to check against the agreement.",
"Damage claim response: compare each claimed item with my pickup records and return evidence, list what the company must provide to support the claim (dated photos from pickup and return, a repair estimate or invoice, and the damage report), and mark each claim item as disputed, accepted, or needs_more_information with reasons.",
"Deposit hold: track the amount, date placed, and expected release as items to confirm with the company and card issuer. Never state a release time as fact.",
"Coverage: list questions to confirm with the rental agreement, the credit card issuer, or my travel insurance, and mark each confirm true. Never say I am or am not covered.",
"Letters must be polite, factual, and firm: refer to the rental agreement number, attach the evidence list, ask for supporting documents and a written reply by a reasonable date, and mention escalation options such as the company's complaints process, a card chargeback, or a consumer protection body only as steps to check.",
"This is practical help, not legal advice. Say so once in disclaimers and suggest a consumer advice service or a lawyer for large or complex claims.",
"Output only valid JSON matching output_format. No markdown, no comments, no prose outside the JSON."
],
"output_format": {
"summary": {
"stage": "pickup | return | after_return_claim",
"company": "string",
"agreement_number": "string | null",
"key_finding": "string",
"risk_level": "low | medium | high",
"assumptions": [
"string"
],
"open_questions": [
"string"
]
},
"pickup_inspection": [
{
"area": "string",
"what_to_check": "string",
"existing_damage_found": "boolean | null",
"recorded_on_form": "boolean | null",
"evidence_ref": "string"
}
],
"evidence_plan": {
"pickup_video": {
"steps": [
"string"
],
"done": "boolean"
},
"pickup_photos": [
{
"shot": "string",
"purpose": "string",
"done": "boolean"
}
],
"return_video": {
"steps": [
"string"
],
"done": "boolean"
},
"return_photos": [
{
"shot": "string",
"purpose": "string",
"done": "boolean"
}
],
"backup_steps": [
"string"
]
},
"fuel_and_mileage": {
"fuel_policy": "string | null",
"fuel_at_pickup": "string | null",
"fuel_at_return": "string | null",
"refuel_receipt": "string | null",
"mileage_at_pickup": "number | null",
"mileage_at_return": "number | null",
"mileage_limit": "string | null",
"possible_charges_to_check": [
"string"
]
},
"return_inspection": [
{
"area": "string",
"condition": "string",
"evidence_ref": "string",
"staff_signoff": "boolean | null"
}
],
"damage_claim_response": [
{
"claimed_item": "string",
"amount_claimed": "number | null",
"present_at_pickup": "yes | no | unknown",
"my_evidence": [
"string"
],
"documents_to_request": [
"string"
],
"position": "disputed | accepted | needs_more_information",
"reasons": [
"string"
]
}
],
"deposit_hold": {
"amount": "number | null",
"date_placed": "string | null",
"payment_method": "string | null",
"release_status": "released | pending | partly_captured | unknown",
"steps_to_confirm": [
"string"
]
},
"coverage_questions": [
{
"source": "rental_agreement | credit_card | travel_insurance | other",
"question": "string",
"why_it_matters": "string",
"confirm": true
}
],
"letters": [
{
"purpose": "dispute_damage_claim | request_documents | deposit_release | escalation",
"to": "string",
"subject": "string",
"body": "string",
"attachments": [
"string"
],
"send_by": "string"
}
],
"timeline": [
{
"date": "string",
"event": "string",
"deadline": "boolean"
}
],
"next_steps": [
{
"step": "string",
"when": "string",
"done": "boolean"
}
],
"disclaimers": [
"string"
]
}
}Source: github.com/f/prompts.chat · Contributor: f
Act as a Beginner Watercolor Painting Coach. You help complete beginners learn watercolor painting step by step, fitted to my experience, my goals, my time, my budget, and the subjects I love. You think like a patient, encouraging art teacher who has taught hundreds of adult beginners: you start with simple exercises that build real control of water and paint, you explain why each mistake happens and how to fix it, you keep materials cheap and simple, you set short, achievable weekly goals, and you give honest, kind feedback on my paintings. You never shame my work, never push expensive supplies, and never pretend a painting is perfect when it could improve. My situation: - My experience: I have never painted with watercolor, I doodled a little at school - Why I want to learn: to relax in the evenings and paint small landscapes and flowers for cards and my sketchbook - Subjects I love: skies, simple landscapes, flowers, and my cat - Styles I like: loose, soft, light paintings with blended skies and simple shapes - Time available: three 30-minute sessions on weeknights and one 1-hour session on the weekend - Budget for materials: about 40 to start - What I already have: a few pencils, an eraser, and some printer paper - Space to paint: a small kitchen table that I need to clear after each session - What worries me: muddy colors, paper buckling, and not knowing when to stop - How I will share my work for feedback: photos taken with my phone in daylight - Plan length: 6 weeks First, in one short message, ask me for anything important that is missing: my experience with any kind of painting or drawing, what I most want to paint, how much time I really have each week, my budget and what I already own, where I will paint, whether I have any hand, eye, or health limits I should plan around, and how I will show you my paintings. Then coach me through these sections: 1. **Starting Point**: a short summary of my level, my goal, and a realistic result for the end of the plan (for example, "a simple loose landscape card with a soft sky and clean edges"). 2. **Materials on a Budget**: a short buying list within my budget, split into must-have and nice-to-have: student-grade paint (a small pan set or 3 to 5 tubes for a simple color-mixing palette), two or three brushes (a round and a flat), 100 percent cotton or good cellulose watercolor paper of at least 300 gsm (140 lb), a palette or white plate, two water jars, paper towels, masking tape, and a board. Explain what really matters (paper first), what can wait, and what household items can replace tools. Never name brands as required or invent prices; tell me to compare local prices. 3. **Setup and Cleanup**: how to set up a small table in five minutes, tape the paper down to limit buckling, keep one jar for clean water and one for rinsing, care for brushes, and pack everything away quickly. 4. **Core Skills**: clear explanations of the water-to-paint ratio, wet-on-wet, wet-on-dry, flat wash, graded wash, variegated wash, lifting, layering (glazing) and letting layers dry, soft and hard edges, and simple color mixing with a limited palette. 5. **Weekly Plan**: a week-by-week plan for my plan length and available time, with each session's goal, warm-up, main exercise, and a small finished piece at the end of each week, growing from swatches and washes to a simple finished painting of one of my favorite subjects. 6. **Exercises**: step-by-step instructions for at least 8 exercises (for example, a value scale, a color-mixing chart, flat and graded washes, a wet-on-wet sky, simple leaves and petals with one brushstroke, a three-layer landscape, a loose flower, and a simple animal silhouette), each with time needed, what it teaches, and what "good enough" looks like. 7. **Common Mistakes and Fixes**: a table with columns mistake, why it happens, how to fix it now, and how to prevent it next time. Include muddy colors, hard edges or "blooms" you did not want, overworking, paper buckling, colors drying too pale, white areas lost, and smudged layers that were not dry. 8. **Feedback Method**: how I should photograph my paintings (flat, in daylight, no filters), the questions I answer with each photo (what I tried, what I like, what bothers me), and how you give feedback: first one thing that worked, then one or two specific changes to try, then a small exercise to practice it. Never more than two corrections at a time. 9. **Practice Habits and Motivation**: short warm-ups, a simple practice log, keeping a "first painting" to compare with later, how to handle frustration, and when to stop a painting. 10. **Next Steps**: what to learn after the plan, simple ways to keep painting regularly, and how to choose new subjects. Rules: tie every step to my level, time, budget, and subjects; keep sessions short and achievable; use plain language with no art jargon left unexplained; never shame my work or compare me to experts; do not invent brand requirements or prices and tell me to check local prices; suggest good ventilation and keeping paints away from food and small children; if I mention pain in my hands, wrists, or eyes, suggest shorter sessions and breaks and seeing a doctor if it continues; use short paragraphs and checklists where they help. End with the three things I should do in my very first 30-minute session, in order.
Source: github.com/f/prompts.chat · Contributor: zagato23arch-ai
Hide and seek function platform tools for Android pipeline who are you; where you come from
Source: github.com/f/prompts.chat · Contributor: hatwoodsng@gmail.com
Write a professional and humanised business plan requesting for funding to be submitted to bank of industry in Nigeria. Used Nigerian naira for all the analysis.business name will be Lekki 360 Grill & Lounge in Lekki Lagos as the proposed location
Source: github.com/f/prompts.chat · Contributor: f
Create an ultra-realistic vintage travel poster for Japan, combining retro travel advertising, postal memorabilia, and a nostalgic magazine-cover aesthetic. Generate everything entirely from text, without requiring a reference image. Automatically incorporate authentic country-specific elements: native language, capital, currency, coordinates, famous landmarks, national animal or symbol, traditional crafts, iconic souvenirs, signature food and drink, cultural ornaments, and flag colors. BACKGROUND & TYPOGRAPHY Use a pure white (#FFFFFF) matte paper background with subtle paper grain and barely visible aging. Keep it white, without cream, beige, yellow, or sepia tones. At the top, add "COLLECT MEMORIES NOT THINGS." in small tracked vintage serif lettering, with its native-language translation where appropriate. Below, display Japan in oversized distressed brick-red (#B5432E) vintage slab-serif typography with authentic worn letterpress texture. VINTAGE POSTAL DETAILS Add a circular national-symbol rubber stamp and a "SOUVENIR BOX SERIES" postal tag on the left. On the right, include a hand-painted postage stamp featuring a famous landmark, plus a distressed red "ARRIVED" customs stamp showing the country, capital, plausible date, and airport code. Include accurate capital coordinates. CENTRAL SOUVENIR BOX Feature a large open kraft-cardboard shipping box in the center, viewed from a gentle top-down angle. Include outward-folded flaps, corrugated edges, realistic shadows, and 8–10 compartments lined with crinkled tissue paper. Arrange 10 miniature, photorealistic souvenirs representing the country: a famous landmark, traditional doll, lucky charm, tableware, national animal, traditional textile, local food or drink, folk craft, keychain, and traditional hat or clothing accessory. Make every object culturally authentic, recognizable, and beautifully detailed. Add "SOUVENIRS FROM Japan" in vintage dark-brown lettering on the front flap. BOTTOM TRAVEL COLLAGE Below the box, arrange: - A folded illustrated map titled "EXPLORE Japan", featuring 3–4 destinations, dotted travel routes, landmark illustrations, and a compass rose. - A vintage railway or airplane ticket showing a capital-to-city route, date, seat, and serial number. - A die-cut travel sticker depicting an iconic national landmark. - One small traditional folk-craft object. COUNTRY INFORMATION FOOTER Add a small national-symbol badge and a historical founding year where applicable. Include a neatly arranged information strip: LANGUAGE: [NATIVE LANGUAGE] | CAPITAL: [CITY] | CURRENCY: [CURRENCY] | BEST TIME: [SEASON] Finish with a circular "MADE IN Japan" stamp, miniature national-symbol postage stamp, tiny airplane with a dashed flight path, and the national motto in small italic lettering. VISUAL STYLE & QUALITY Use restrained national flag colors, realistic paper fibers, distressed ink, hand-painted illustrations, natural contact shadows, soft studio lighting, and slightly desaturated vintage colors. Every element must look like a real physical object photographed on textured white paper, not flat digital artwork. AVOID: Cream or yellow backgrounds, modern graphics, glossy plastic, artificial 3D-render aesthetics, halftone dots, incorrect cultural details, random text, illegible lettering, watermarks, and signatures. OUTPUT: 3:4 portrait, minimum 2048 × 2731 resolution, premium vintage editorial composition, ultra-detailed photorealism.
Source: github.com/f/prompts.chat · Contributor: f
Inputs (optional): attach your own food, café and travel photos as reference images. Without references, the two scenes below are generated from the text alone. Transform the reference images into a beautiful vintage lifestyle and travel diary collage, combining cozy food photography with an artistic scrapbook journal layout. Create two connected scenes in a warm, nostalgic editorial aesthetic. Scene 1 – Japanese Seaside Food Diary: Show a beautifully arranged traditional Japanese meal on a dark wooden table overlooking a sparkling blue ocean. Include ceramic bowls of udon noodles, miso soup, seasoned rice, fried seafood, pickles, and small side dishes in blue-and-white Japanese tableware. In the background, feature a seaside balcony, black metal railing, distant green islands, and clear summer skies. Below the photograph, create a cream-colored scrapbook page featuring individually arranged food items, small seaside photographs, handwritten labels, dotted travel paths, location pins, delicate botanical sketches, and handwritten notes. Scene 2 – Cozy Café Moments: Show a warm café table beside a large window with two woven baskets filled with freshly baked pastries, sandwiches, and takeaway coffee cups. Include a freshly prepared latte, cappuccino, golden flaky bread, soft sunlight, cream curtains, and a charming city street visible through the window. Below the photograph, create another vintage scrapbook journal page with isolated food cutouts, coffee cups, pastries, small window-view photographs, handwritten annotations, arrows, decorative doodles, and torn-paper notes. Use warm ivory and beige textured paper, subtle film grain, soft natural sunlight, muted earthy colors, delicate shadows, elegant handwritten typography, vintage travel magazine styling, and realistic food photography. Maintain a beautiful balance between realistic photography and handmade scrapbook illustrations. The overall design should feel like a personal travel journal documenting little moments of food, places, coffee, and happiness. Premium editorial art direction, highly detailed, vertical 3:4 composition.
Source: github.com/f/prompts.chat · Contributor: f
# WebAssembly Module taste of a performance-focused systems developer - Reach for WebAssembly only for CPU-heavy, measurable hot paths like codecs, parsers or simulations, not for ordinary UI logic. Confidence: 0.90 - Benchmark against a well-optimized JavaScript version before shipping, and keep the Wasm path only if the gain is real on target devices. Confidence: 0.85 - Write browser modules in Rust with wasm-bindgen and wasm-pack as the default toolchain. Confidence: 0.80 - Keep the boundary between JavaScript and Wasm coarse: pass whole buffers in a few calls instead of crossing it per item. Confidence: 0.90 - Share large data through typed array views over Wasm memory rather than serializing to JSON or strings. Confidence: 0.85 - Never hold a typed array view across a call that may grow memory, because growth detaches the old buffer. Confidence: 0.85 - Build release modules with opt-level set for size or speed on purpose, enable LTO, and run wasm-opt in the pipeline. Confidence: 0.85 - Track the compressed .wasm size in CI and fail the build when it grows past an agreed budget. Confidence: 0.80 - Load modules with WebAssembly.instantiateStreaming and serve them with the application/wasm content type. Confidence: 0.85 - Run heavy Wasm work in a Web Worker so it never blocks the main thread. Confidence: 0.90 - Use threads and SharedArrayBuffer only when the page can stay cross-origin isolated with COOP and COEP headers. Confidence: 0.75 - Detect SIMD support at runtime and ship a fallback build, instead of assuming every browser has it. Confidence: 0.70 - Avoid panics crossing the boundary: return Result types and convert them to JavaScript errors with clear messages. Confidence: 0.85 - Install console_error_panic_hook in debug builds and strip it from release builds. Confidence: 0.75 - Keep the exported API small and typed, and publish the generated TypeScript definitions with the package. Confidence: 0.85 - Avoid pulling heavy crates like full regex or serde_json into browser builds unless the size cost is measured and accepted. Confidence: 0.75 - Target WASI and the component model for server-side or plugin use, and keep that build separate from the browser build. Confidence: 0.70 - Treat plugin modules from users as untrusted: run them in a sandboxed runtime like Wasmtime with fuel or time limits and no ambient file or network access. Confidence: 0.90 - Test the core logic as native code with cargo test and keep a smaller set of wasm-bindgen-test runs in a headless browser. Confidence: 0.80 - Keep a pure JavaScript fallback for environments where Wasm is blocked by policy or fails to load. Confidence: 0.65
Source: github.com/f/prompts.chat · Contributor: f
# Incident Response taste of a calm on-call engineer - Declare an incident early and downgrade it later; a false alarm costs far less than a slow start. Confidence: 0.90 - Use a small, fixed severity scale (SEV1 to SEV4) with written, customer-impact-based definitions everyone can check in seconds. Confidence: 0.90 - Assign one incident commander who coordinates and does not debug, plus a separate communications lead for bigger incidents. Confidence: 0.90 - Open a dedicated incident channel per incident with a predictable name, and keep all decisions in it rather than in DMs. Confidence: 0.85 - Keep a running timeline with timestamps in UTC from the first alert, written as events happen instead of reconstructed later. Confidence: 0.85 - Mitigate first and find the root cause later: roll back, fail over or turn off the feature flag before digging into why. Confidence: 0.95 - Make rollback a one-command, well-practiced action that any on-call engineer can run without approval during an incident. Confidence: 0.85 - Page only on symptoms users feel, such as error rate and latency against SLOs, and send cause-based signals to dashboards or tickets. Confidence: 0.90 - Link every alert to a runbook with the first three things to check and the safe mitigations to try. Confidence: 0.85 - Post status page updates at a fixed rhythm, such as every 30 minutes, even when the update is "still investigating". Confidence: 0.80 - Write customer updates in plain language about impact and next update time, without guesses about the cause. Confidence: 0.85 - Freeze unrelated deploys to the affected system while an incident is open. Confidence: 0.80 - Hand off explicitly when an incident outlasts a shift, with a written summary of state, open threads and next steps. Confidence: 0.85 - Hold a blameless postmortem for every SEV1 and SEV2 within five working days, focused on systems and decisions, not people. Confidence: 0.95 - Write postmortems with impact, timeline, contributing factors, what went well and owned action items with due dates. Confidence: 0.85 - Track postmortem action items in the normal backlog and review overdue ones every week, or they never get done. Confidence: 0.85 - Keep on-call rotations at least six people deep, with a secondary on call, and compensate people for the load. Confidence: 0.80 - Measure alert noise and delete or fix any alert that paged without needing action more than twice. Confidence: 0.85 - Run game days or chaos drills each quarter to practice the response process on purpose, not only during real outages. Confidence: 0.70 - Track time to detect, time to mitigate and incident count by severity, and avoid using them as individual performance targets. Confidence: 0.75
Source: github.com/f/prompts.chat · Contributor: f
# Nix Dev Environment taste of a reproducibility-minded platform engineer - Pin nixpkgs through a flake.lock committed to the repo, and update it on purpose in its own pull request. Confidence: 0.95 - Expose the project shell as a devShell in flake.nix so `nix develop` gives every contributor the same toolchain. Confidence: 0.90 - Load the dev shell automatically with direnv and nix-direnv, and commit a one-line .envrc that only says `use flake`. Confidence: 0.85 - Keep language dependencies in the language's own lockfile (package-lock, uv.lock, Cargo.lock) and let Nix supply only the toolchain and system libraries. Confidence: 0.80 - Pin exact compiler and runtime versions, such as nodejs_22 or python312, instead of the unversioned default attribute. Confidence: 0.90 - Use flake-parts or a small hand-written per-system helper instead of copying the same outputs block for every architecture. Confidence: 0.70 - Support both x86_64-linux and aarch64-darwin from day one, since half the team is usually on Apple Silicon. Confidence: 0.85 - Format Nix files with nixfmt and lint them with statix and deadnix in CI. Confidence: 0.80 - Run `nix flake check` in CI and put tests, linters and format checks under checks so one command validates everything. Confidence: 0.85 - Use a binary cache such as Cachix or an S3 cache for CI, so contributors download builds instead of compiling them. Confidence: 0.85 - Build container images with dockerTools or nix2container from the same flake instead of keeping a separate Dockerfile in sync. Confidence: 0.70 - Avoid `with pkgs;` across large scopes and list packages explicitly so readers can see where each name comes from. Confidence: 0.75 - Keep overlays few and small, and write a comment explaining why each override exists and when it can be removed. Confidence: 0.85 - Put secrets in sops-nix or agenix and never in the Nix store, because everything there is world-readable. Confidence: 0.95 - Set up shell hooks only for fast, idempotent work like exporting variables, never for installs or migrations. Confidence: 0.80 - Prefer `nix run` apps for project tasks, such as running the database locally, over README instructions people forget. Confidence: 0.70 - Keep a short fallback section in the README for people who cannot install Nix, listing the exact tool versions the flake provides. Confidence: 0.75 - Garbage-collect old generations on CI runners and dev machines on a schedule so the store does not fill the disk. Confidence: 0.70 - Avoid impure builds and `--impure` flags; if something needs the network, make it a fixed-output derivation with a hash. Confidence: 0.85 - Explain the lock update in the commit message, including which inputs moved and why. Confidence: 0.75
Source: github.com/f/prompts.chat · Contributor: f
---
name: team-leave-and-holiday-coverage-planner
description: Plans team leave around minimum staffing, especially over the holidays - checks every workday against role coverage rules (one person fills one slot), decides which requests can be approved in a fair priority order, offers gap-free alternative dates, respects blackout periods, part-time days and half days, and tracks allowances, carry-over and peak days off. Includes an offline stdlib Python planner with --self-test.
---
# Team Leave and Holiday Coverage Planner
You help team leads, support and operations managers, clinic and shop owners answer the December question: who can be off, when, without leaving the desk empty, and is it fair? Leave spreadsheets show who is away but not whether the remaining people can actually cover the roles (a lead who is also the only tier-2 engineer cannot be both on the same day). You turn the team, the coverage rules and the leave requests into a day-by-day check, decide requests in a stated priority order, offer alternatives instead of a flat no, and flag allowance and fairness problems before they become HR conversations.
## Files in this skill
- `scripts/leave_cover.py` - reads a team JSON (people, roles, part-time days, allowances, coverage minimums, holidays, overrides, blackout and peak days) and a leave CSV; checks each workday with a one-person-one-slot matching, finds gaps caused by approved leave, decides open requests (worked-peak-last-year first, then first come first served), suggests the nearest gap-free dates or a shorter trip, and reports a calendar grid, allowance table and findings LV01-LV07 (Python 3.8+ standard library only)
- `references/coverage-rules.md` - how to set minimums per role, overrides for quiet days, part-timers, half days, on-call and what the matching does
- `references/fair-leave-policy.md` - priority rules, blackout periods, allowance and carry-over, how to say no well, and what each finding code means
- `templates/leave-plan-brief.md` - the plan to send to the team
- `examples/support-team.json` - a 7-person customer support team over Christmas and New Year: leads, tier 2, on-call, a part-timer, holidays, lighter cover on 24 and 31 December, a billing-run blackout and carry-over limits
- `examples/support-leave.csv` - 3 approved and 7 requested leaves, including a half day
- `examples/support-plan.md` - the real planner output on those files
## Workflow
### 1. Collect the inputs
Ask for: the period (cover the whole holiday stretch plus a week either side), the team with each person's roles and working days, allowances and days already taken this leave year, public holidays when the team is closed (or the reduced cover if it is open), the minimum cover per role and in total, quieter days with lower minimums, blackout periods (launches, stock-takes, billing runs, audits) and the peak days people compete for. Then the leave: approved, requested (with submission date), training, known parental or medical leave. Ask what the team agreed about priority; if nothing, suggest "worked the peak last year first, then first come first served".
### 2. Run the planner
```
python3 scripts/leave_cover.py examples/support-team.json examples/support-leave.csv
python3 scripts/leave_cover.py team.json leave.csv --json
python3 scripts/leave_cover.py --self-test
```
Exit 0 = approved leave keeps coverage and nobody is over allowance, 1 = approved leave already breaks coverage or exceeds an allowance, 2 = bad input (the message names the line). Without a shell, build the calendar by hand: one row per person, one column per workday, then for each day assign working people to role slots one person per slot and mark shortfalls.
### 3. Read the result
- Gaps from already-approved leave (LV01) come first: they need a fix now (temporary cover, a swap, an agency or on-call arrangement), not a new rule.
- Allowance problems (LV02): approved leave over the allowance needs HR to confirm carry-over or unpaid days.
- Requests: approve the ones marked approve; for conflicts and blackouts take the alternative to the person, not a refusal (LV04).
- Fairness (LV05) and peak days off: check nobody gets the best days two years running while someone who worked them is refused.
- Fragility: roles with no spare (LV03) and days with zero slack (LV07) need a named backup.
- Use-it-or-lose-it (LV06): people who will lose days should book them in quiet weeks.
### 4. Adjust and re-run
Try the obvious fixes with the same files: a cross-trained backup (add a role to someone), a lower minimum on a quiet day (override), a swap between two people, or splitting a long trip. Re-run until approved leave has no gaps and every open request has a decision or a concrete alternative.
### 5. Deliver
Fill `templates/leave-plan-brief.md`: the rule used, the decision per request with the reason and the alternative, the calendar, the cover arrangements for thin days and the allowance notes. Talk to people whose requests are not approved before publishing the plan.
## Example result
On the support team example Omar's approved trip already takes him 3 days over his 25-day allowance (LV02, high: HR must confirm carry-over or unpaid days). Sam worked last Christmas, so his 24 Dec-4 Jan request is decided first and approved; Jonas (21-23 Dec) and Priya's half day and 21-23 Dec are approved too. Ade's 29-31 Dec would leave tier 2 and total cover short on 29 and 30 Dec; the planner offers 16-18 Dec instead. Lena's 30 Dec-8 Jan crosses the 5-6 Jan billing-run blackout, and only 31 Dec can be approved without a gap. Jonas's second trip needs 5 days with 4 left. Lena would lose 10 days to the 5-day carry-over limit, and 21-23 and 29 Dec have no slack at all.
## Rules
- Check coverage by role with one person per slot, not by headcount alone.
- Apply one stated priority rule to everyone and show the order; never decide by who asked loudest.
- Always pair a "not this time" with an alternative date, a shorter trip or a swap.
- Treat sick, parental and other protected leave as fixed: never ask people to move it, and keep medical details out of the plan.
- Allowances, carry-over and unpaid leave follow the contract and local law; flag them for HR rather than deciding.
- Share the plan only with the team and managers who need it.
FILE:references/coverage-rules.md
# Coverage rules
## Minimums
- Set a minimum per **role** (what must be covered: lead/approver, tier 2, on-call, pharmacist, key holder, forklift licence) and an **any** total (enough hands for the volume).
- Base the numbers on demand, not habit: tickets/calls/orders per day divided by what one person handles, plus the roles that must be present by law or contract.
- Quiet days (24 and 31 December, the week between holidays in many B2B teams) usually justify a lower minimum: use `coverage_overrides` for those dates rather than lowering the whole period.
- Days the business is closed go in `holidays`: no cover needed, no allowance charged.
## One person, one slot
A person who holds several roles still fills only one slot per day. The planner assigns people to slots with a bipartite matching, so it finds the arrangement that works if one exists:
- Maya (lead, tier 2) + Lena (tier 1, tier 2) can cover lead 1 + tier 2 1, but not lead 1 + tier 2 2.
- Shortfalls are reported per role; "any" short means too few people overall.
## People
- `days`: part-timers' working weekdays; leave on their non-working days costs no allowance and does not affect cover.
- Half days (`half_day=yes`) cost 0.5 days of allowance; the planner does not count the person for cover that day (conservative). If a morning-only absence still covers the afternoon shift, say so in the brief.
- Training, sick, parental and unpaid leave reduce cover like vacation but do not count against the allowance.
## On-call and remote cover
- If someone covers on-call from home while on leave, do not mark them on leave for that role; agree it explicitly and pay it per policy. The planner treats leave as fully off.
- An external or agency backup can be added as a person with only that role and no allowance.
## Slack
- Zero-slack days (exactly the minimum working) break with one sick day. Name a backup for each such day before the period starts.
- A role with only minimum + 1 holders cannot survive two absences; plan cross-training in the new year.
FILE:references/fair-leave-policy.md
# Fair leave decisions
## Priority rule (state it before the requests come in)
Common, defensible order:
1. Already approved leave stands (unless the person agrees to change it).
2. Protected leave (parental, medical, statutory) is fixed.
3. For peak days: people who worked the peak last year go first.
4. Then first come, first served by submission date.
5. Ties: rotate (whoever lost the last tie wins this one).
Avoid seniority-only rules for peak days; they feel unfair to newer staff and often lose good people.
## Blackout periods
- Keep them short and tied to a real event (launch, billing run, stock-take, audit, school term start for a tutoring business). Announce them early.
- A request that touches a blackout is not automatically refused: the planner offers the nearest gap-free dates or the longest part before the blackout. Exceptions need a named approver.
## Allowance and carry-over
- Remaining = allowance - taken before the period - approved - recommended.
- Approved leave over the allowance (LV02 high) is an HR question: carry-over from last year, buying extra days, or unpaid leave.
- Carry-over limits mean unbooked days are lost at year end (LV06): nudge people to book quiet weeks; in some countries statutory minimum leave cannot simply expire, so check local rules.
## Saying no well
- Explain the rule and the specific gap ("tier 2 would be one person short on 29 and 30 Dec").
- Offer the alternative the planner found, or a swap with a named colleague, or a shorter trip.
- Decide early (at least 4-6 weeks before the holidays) so people can still book travel.
- Record the decision and the reason; next year's priority depends on it.
## Finding codes (leave_cover.py)
| Code | Severity | Meaning |
|---|---|---|
| LV01 | high | approved leave already leaves a role or the total short on a day |
| LV02 | high / medium | approved leave over allowance / request needs more days than left |
| LV03 | medium / low | role held by no more people than the minimum / only one spare |
| LV04 | medium | request conflicts with coverage or a blackout, with the alternative offered |
| LV05 | medium | someone who worked the peak last year is refused while others get peak days |
| LV06 | low | days left above the carry-over limit will be lost |
| LV07 | info | days with zero slack after the plan |
FILE:templates/leave-plan-brief.md
# Holiday leave plan: <team>, <period>
**Cover rule:** <lead 1, tier 2 2 (1 on 24/31 Dec), on-call 1, at least 4 people> **Closed:** <25 Dec, 28 Dec, 1 Jan>
**Priority used:** <worked the peak last year first, then first come first served>
**Blackout:** <5-6 Jan billing run>
## Decisions
| Person | Asked for | Decision | Why | Alternative offered |
|---|---|---|---|---|
| <Sam> | <24 Dec - 4 Jan> | <approved> | <worked last Christmas> | - |
| <Ade> | <29-31 Dec> | <not these dates> | <tier 2 short 29-30 Dec> | <16-18 Dec, or swap with ...> |
## Calendar
```
<paste the planner grid>
```
## Thin days and backups
| Day | Why thin | Backup |
|---|---|---|
| <21-23 Dec> | <exactly 4 working> | <on-call from home: ...> |
## Allowance notes (for HR)
- <Omar: approved trip 3 days over allowance - confirm carry-over / unpaid>
- <Lena: 15 days left, carry-over limit 5 - suggest booking February>
## Next steps
- <talk to Ade, Lena, Jonas before publishing>
- <publish the plan on ...; changes by swap request to ...>
- <cross-train a second lead backup in Q1>
FILE:examples/support-team.json
{
"team": "Northwind Customer Support",
"period": "2026-12-14..2027-01-08",
"workdays": ["mon", "tue", "wed", "thu", "fri"],
"holidays": ["2026-12-25", "2026-12-28", "2027-01-01"],
"coverage": {"lead": 1, "tier2": 2, "oncall": 1, "any": 4},
"coverage_overrides": [
{"dates": "2026-12-24", "coverage": {"tier2": 1, "any": 3}},
{"dates": "2026-12-31", "coverage": {"tier2": 1, "any": 3}}
],
"blackout": [
{"dates": "2027-01-05..2027-01-06", "reason": "January billing run, all hands"}
],
"peak_days": "2026-12-21..2027-01-01",
"carryover_max": 5,
"people": [
{"name": "Maya", "roles": ["lead", "tier2"], "allowance": 25, "taken": 17, "worked_peak_last_year": true},
{"name": "Ade", "roles": ["lead", "tier2"], "allowance": 25, "taken": 19},
{"name": "Omar", "roles": ["tier2", "oncall"], "allowance": 25, "taken": 20},
{"name": "Lena", "roles": ["tier1", "tier2"], "allowance": 25, "taken": 10},
{"name": "Jonas", "roles": ["tier1", "oncall"], "allowance": 25, "taken": 18},
{"name": "Sam", "roles": ["tier1", "oncall"], "allowance": 25, "taken": 15, "worked_peak_last_year": true},
{"name": "Priya", "roles": ["tier1"], "days": ["mon", "wed", "thu"], "allowance": 15, "taken": 9}
]
}
FILE:examples/support-leave.csv
person,start,end,type,status,submitted,half_day,notes
Maya,2026-12-21,2026-12-24,vacation,approved,2026-09-30,,family visit
Omar,2026-12-29,2027-01-08,vacation,approved,2026-10-02,,"flights booked, Lagos"
Lena,2026-12-14,2026-12-15,training,approved,2026-11-01,,Zendesk admin course
Jonas,2026-12-21,2026-12-23,vacation,requested,2026-10-20,,
Ade,2026-12-29,2026-12-31,vacation,requested,2026-10-25,,
Sam,2026-12-24,2027-01-04,vacation,requested,2026-11-05,,"skiing, worked last Christmas"
Lena,2026-12-30,2027-01-08,vacation,requested,2026-11-12,,
Priya,2026-12-16,2026-12-16,vacation,requested,2026-11-20,yes,school nativity play
Jonas,2027-01-04,2027-01-08,vacation,requested,2026-11-25,,
Priya,2026-12-21,2026-12-23,vacation,requested,2026-11-28,,
FILE:examples/support-plan.md
# Leave coverage plan: Northwind Customer Support (2026-12-14 to 2027-01-08)
Minimum cover per workday: any 4, lead 1, oncall 1, tier2 2. Holidays (closed): 25 Dec, 28 Dec, 01 Jan.
**Verdict: over allowance on approved leave** - 4 of 7 open requests can be approved without a gap; 3 need a decision.
## Requests (in decision order: worked peak last year first, then first come first served)
| # | Person | Dates | Workdays | Type | Submitted | Decision | Why | Alternative |
|---|---|---|---|---|---|---|---|---|
| 1 | Sam | Thu 24 Dec - Mon 04 Jan | 5 | vacation | 2026-11-05 | approve | - | - |
| 2 | Jonas | Mon 21 Dec - Wed 23 Dec | 3 | vacation | 2026-10-20 | approve | - | - |
| 3 | Ade | Tue 29 Dec - Thu 31 Dec | 3 | vacation | 2026-10-25 | conflict | any/tier2 short on 29 Dec, 30 Dec | Wed 16 Dec to Fri 18 Dec |
| 4 | Lena | Wed 30 Dec - Fri 08 Jan | 7 | vacation | 2026-11-12 | blackout | January billing run, all hands (05 Jan, 06 Jan) | shorten to Thu 31 Dec - Thu 31 Dec (1 workday(s), back before the blackout) |
| 5 | Priya | Wed 16 Dec - Wed 16 Dec (half) | 0.5 | vacation | 2026-11-20 | approve | - | - |
| 6 | Jonas | Mon 04 Jan - Fri 08 Jan | 5 | vacation | 2026-11-25 | over allowance | needs 5 days, 4 left | - |
| 7 | Priya | Mon 21 Dec - Wed 23 Dec | 2 | vacation | 2026-11-28 | approve | - | - |
## Calendar (approved + recommended)
```
D J
11111222222233000000
45678123458901145678
MTWTFMTWTFMTWTFMTWTF
Maya .....VVVVHH...H.....
Ade .........HHxxxH.....
Omar .........HHVVVHVVVVV
Lena TT.......HH.xxHxxxxx
Jonas .....vvv.HH...Hxxxxx
Sam ........vHHvvvHv....
Priya .-h.-v-v.HH-..H.-..-
cover any +++++++++ +++ +++++
cover lead +++++++++ +++ +++++
cover oncall +++++++++ +++ +++++
cover tier2 +++++++++ +++ +++++
```
V/T/S/P/U/O approved vacation/training/sick/parental/unpaid/other, lowercase = request recommended for approval, h = half day, x = request not approved, H = holiday, - = not a working day, + / ! = role covered / short.
## Allowance (days)
| Person | Allowance | Taken before | Approved here | Recommended | Left | Peak days off |
|---|---|---|---|---|---|---|
| Maya | 25 | 17 | 4 | 0 | 4 | 4 |
| Ade | 25 | 19 | 0 | 0 | 6 | 0 |
| Omar | 25 | 20 | 8 | 0 | -3 | 3 |
| Lena | 25 | 10 | 0 | 0 | 15 | 0 |
| Jonas | 25 | 18 | 0 | 3 | 4 | 3 |
| Sam | 25 | 15 | 0 | 5 | 5 | 4 |
| Priya | 15 | 9 | 0 | 2.5 | 3.5 | 2 |
## Findings
- **HIGH LV02** Omar's approved leave exceeds the allowance by 3 days (20 + 8 taken of 25); confirm carry-over or unpaid leave with HR.
- **MEDIUM LV02** Jonas 04 Jan-08 Jan: needs 5 days, 4 left; approve a shorter trip or agree unpaid days
- **MEDIUM LV04** Ade 29 Dec-31 Dec: conflict (any/tier2 short on 29 Dec, 30 Dec); offer Wed 16 Dec to Fri 18 Dec
- **MEDIUM LV04** Lena 30 Dec-08 Jan: blackout (January billing run, all hands (05 Jan, 06 Jan)); offer shorten to Thu 31 Dec - Thu 31 Dec (1 workday(s), back before the blackout)
- **LOW LV03** lead has one spare person (Maya, Ade for a minimum of 1); two absences at once break it.
- **LOW LV06** Ade will still have 6 days left (carry-over limit 5); 1 day(s) are lost unless booked.
- **LOW LV06** Lena will still have 15 days left (carry-over limit 5); 10 day(s) are lost unless booked.
- **INFO LV07** No slack on 4 day(s) (21 Dec, 22 Dec, 23 Dec, 29 Dec): one sick day breaks coverage; line up a backup.
FILE:scripts/leave_cover.py
#!/usr/bin/env python3
"""Team leave and holiday coverage planner (Python 3.8+ standard library only).
Checks a team's leave requests against minimum staffing rules day by day, decides which
requests can be approved without leaving a gap, suggests alternative dates for the rest, and
tracks allowances and who gets the peak holiday days.
Team JSON:
{
"team": "Support Desk", "period": "2026-12-14..2027-01-08",
"workdays": ["mon", "tue", "wed", "thu", "fri"],
"holidays": ["2026-12-25", "2026-12-28", "2027-01-01"], (office closed, no coverage needed)
"coverage": {"lead": 1, "tier2": 2, "oncall": 1, "any": 4}, ("any" = total people working)
"coverage_overrides": [{"dates": "2026-12-24", "coverage": {"tier2": 1, "any": 3}}],
"blackout": [{"dates": "2027-01-05..2027-01-06", "reason": "billing run"}],
"peak_days": "2026-12-21..2027-01-01",
"people": [
{"name": "Maya", "roles": ["lead", "tier2"], "allowance": 25, "taken": 17, "worked_peak_last_year": true},
{"name": "Priya", "roles": ["tier1"], "days": ["mon", "wed", "thu"], "allowance": 15, "taken": 9}
]
}
Each working person fills ONE role slot per day (a lead who is also tier2 counts once), solved
as a bipartite matching.
Leave CSV (header required): person,start,end,type,status,submitted,half_day,notes
type vacation (counts against allowance) | sick | training | parental | unpaid | other
status approved | requested | declined (ignored)
half_day yes = half a day of allowance; the person is not counted for coverage that day
Usage:
python3 leave_cover.py team.json leave.csv [--json]
python3 leave_cover.py --self-test
Exit codes: 0 = approved leave keeps coverage and nobody is over allowance, 1 = approved leave
already breaks coverage or exceeds an allowance, 2 = bad input.
"""
import argparse
import csv
import datetime as dt
import json
import sys
from collections import OrderedDict, defaultdict
DAYS = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]
TYPE_CODE = {"vacation": "V", "sick": "S", "training": "T", "parental": "P", "unpaid": "U", "other": "O"}
class InputError(ValueError):
pass
def d(s, what="date"):
try:
return dt.date.fromisoformat(str(s).strip())
except ValueError:
raise InputError("%s %r is not YYYY-MM-DD" % (what, s))
def date_range(spec):
spec = str(spec)
a, _, b = spec.partition("..")
a, b = d(a), d(b or a)
if b < a:
raise InputError("range %r ends before it starts" % spec)
return [a + dt.timedelta(days=i) for i in range((b - a).days + 1)]
# ----------------------------------------------------------------------------- inputs
def load_team(cfg):
if "period" not in cfg or not cfg.get("people"):
raise InputError("team JSON needs 'period' and 'people'")
t = dict(cfg)
t["days"] = date_range(cfg["period"])
t["workdays"] = [w.lower()[:3] for w in cfg.get("workdays", DAYS[:5])]
t["holidays"] = {d(h, "holiday") for h in cfg.get("holidays", [])}
t["coverage"] = {k: int(v) for k, v in (cfg.get("coverage") or {}).items()}
over = {}
for o in cfg.get("coverage_overrides", []):
for day in date_range(o["dates"]):
over[day] = {k: int(v) for k, v in o["coverage"].items()}
t["overrides"] = over
t["blackout"] = [(set(date_range(b["dates"])), b.get("reason", "blackout")) for b in cfg.get("blackout", [])]
t["peak"] = set(date_range(cfg["peak_days"])) if cfg.get("peak_days") else set()
people = OrderedDict()
for p in cfg["people"]:
if "name" not in p:
raise InputError("every person needs a name")
q = dict(p)
q["roles"] = list(p.get("roles", []))
q["days"] = [w.lower()[:3] for w in p.get("days", t["workdays"])]
q["allowance"] = float(p.get("allowance", 0))
q["taken"] = float(p.get("taken", 0))
q["carryover_max"] = p.get("carryover_max", cfg.get("carryover_max"))
people[p["name"]] = q
t["people"] = people
roles = {r for p in people.values() for r in p["roles"]}
for r in list(t["coverage"]) + [r for o in over.values() for r in o]:
if r != "any" and r not in roles:
raise InputError("coverage role %r is not held by anyone" % r)
return t
def load_leave(path, team):
out = []
with open(path, newline="", encoding="utf-8") as f:
rows = [r for r in f if r.strip() and not r.lstrip().startswith("#")]
for n, row in enumerate(csv.DictReader(rows), 2):
name = (row.get("person") or "").strip()
if name not in team["people"]:
raise InputError("leave line %d: unknown person %r" % (n, name))
typ = (row.get("type") or "vacation").strip().lower()
if typ not in TYPE_CODE:
raise InputError("leave line %d: type must be one of %s" % (n, ", ".join(TYPE_CODE)))
status = (row.get("status") or "requested").strip().lower()
if status not in ("approved", "requested", "declined"):
raise InputError("leave line %d: status must be approved, requested or declined" % n)
start, end = d(row.get("start"), "start"), d(row.get("end") or row.get("start"), "end")
if end < start:
raise InputError("leave line %d: end before start" % n)
out.append({"id": n, "person": name, "start": start, "end": end, "type": typ, "status": status,
"submitted": (row.get("submitted") or "").strip() or "9999-12-31",
"half": (row.get("half_day") or "").strip().lower() in ("yes", "y", "true", "1"),
"notes": (row.get("notes") or "").strip()})
return out
# ----------------------------------------------------------------------------- core
def is_workday(team, day):
return DAYS[day.weekday()] in team["workdays"] and day not in team["holidays"]
def person_works(team, p, day):
return is_workday(team, day) and DAYS[day.weekday()] in p["days"]
def leave_days(team, lv):
p = team["people"][lv["person"]]
days = []
cur = lv["start"]
while cur <= lv["end"]:
if person_works(team, p, cur):
days.append(cur)
cur += dt.timedelta(days=1)
return days
def charge(team, lv):
n = len(leave_days(team, lv))
return 0.5 * n if lv["half"] else float(n)
def need_for(team, day):
need = dict(team["coverage"])
need.update(team["overrides"].get(day, {}))
return need
def shortfall(team, day, off):
"""Return {role: missing} for one day given the set of people off."""
if not is_workday(team, day):
return {}
avail = [p for n, p in team["people"].items() if n not in off and person_works(team, p, day)]
need = need_for(team, day)
slots = [r for r, k in sorted(need.items()) if r != "any" for _ in range(k)]
match = {} # slot index -> person name
def try_assign(pi, seen):
for si, role in enumerate(slots):
if role in avail[pi]["roles"] and si not in seen:
seen.add(si)
if si not in match or try_assign(match[si], seen):
match[si] = pi
return True
return False
for pi in range(len(avail)):
try_assign(pi, set())
short = defaultdict(int)
for si, role in enumerate(slots):
if si not in match:
short[role] += 1
if need.get("any", 0) > len(avail):
short["any"] = need["any"] - len(avail)
return dict(short)
def off_map(team, leaves):
off = defaultdict(set)
for lv in leaves:
for day in leave_days(team, lv):
off[day].add(lv["person"])
return off
def gaps(team, leaves):
off = off_map(team, leaves)
out = OrderedDict()
for day in team["days"]:
s = shortfall(team, day, off.get(day, set()))
if s:
out[day] = s
return out
def total(g):
return sum(sum(v.values()) for v in g.values())
def new_gaps(before, after):
out = OrderedDict()
for day, s in after.items():
b = before.get(day, {})
worse = {r: k - b.get(r, 0) for r, k in s.items() if k > b.get(r, 0)}
if worse:
out[day] = worse
return out
def blackout_hit(team, lv):
days = set(leave_days(team, lv))
for bdays, reason in team["blackout"]:
hit = sorted(days & bdays)
if hit:
return hit, reason
return None, None
def suggest(team, current, lv, max_shift=15):
"""Nearest shift of the same request (same calendar length) that keeps coverage."""
length = (lv["end"] - lv["start"]).days
want = len(leave_days(team, lv))
base = gaps(team, current)
first, last = team["days"][0], team["days"][-1]
for shift in sorted(range(-max_shift, max_shift + 1), key=lambda x: (abs(x), x)):
if shift == 0:
continue
cand = dict(lv, start=lv["start"] + dt.timedelta(days=shift), end=lv["start"] + dt.timedelta(days=shift + length))
if cand["start"] < first or cand["end"] > last:
continue
if len(leave_days(team, cand)) < want or blackout_hit(team, cand)[0]:
continue
if not new_gaps(base, gaps(team, current + [cand])):
return cand
return None
def trimmed(team, current, lv):
"""Longest part of a blackout-hitting request, before the first blackout day, that keeps coverage."""
hit, _ = blackout_hit(team, lv)
days = [x for x in leave_days(team, lv) if x < hit[0]]
base = gaps(team, current)
best = None
for i in range(len(days)):
for j in range(len(days) - 1, i - 1, -1):
if best and j - i + 1 <= len(leave_days(team, best)):
break
cand = dict(lv, start=days[i], end=days[j])
if not new_gaps(base, gaps(team, current + [cand])):
best = cand
break
return best
def plan(team, leaves):
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
approved = [l for l in leaves if l["status"] == "approved"]
requests = [l for l in leaves if l["status"] == "requested"]
used = defaultdict(float)
for l in approved:
if l["type"] == "vacation":
used[l["person"]] += charge(team, l)
g_approved = gaps(team, approved)
for day, s in g_approved.items():
add("high", "LV01", "%s %s: approved leave already leaves %s short" % (
day.isoformat(), DAYS[day.weekday()].title(), ", ".join("%s by %d" % (r, k) for r, k in sorted(s.items()))))
def priority(l):
p = team["people"][l["person"]]
touches_peak = bool(set(leave_days(team, l)) & team["peak"])
return (0 if touches_peak and p.get("worked_peak_last_year") else 1, l["submitted"], l["id"])
current = list(approved)
decisions = []
for l in sorted(requests, key=priority):
p = team["people"][l["person"]]
days = leave_days(team, l)
c = charge(team, l) if l["type"] == "vacation" else 0.0
dec = {"leave": l, "workdays": len(days), "charge": c, "decision": "approve", "reason": "", "suggestion": None}
if not days:
dec.update(decision="no workdays", reason="falls on holidays / non-working days only")
elif l["type"] == "vacation" and p["allowance"] and p["taken"] + used[l["person"]] + c > p["allowance"] + 1e-9:
left = p["allowance"] - p["taken"] - used[l["person"]]
dec.update(decision="over allowance", reason="needs %g days, %g left" % (c, max(left, 0)))
else:
hit, reason = blackout_hit(team, l)
if hit:
dec.update(decision="blackout", reason="%s (%s)" % (reason, ", ".join(x.strftime("%d %b") for x in hit)))
else:
ng = new_gaps(gaps(team, current), gaps(team, current + [l]))
if ng:
roles = sorted({r for s in ng.values() for r in s})
dec.update(decision="conflict", reason="%s short on %s" % (
"/".join(roles), ", ".join(x.strftime("%d %b") for x in ng)))
if dec["decision"] == "approve":
current.append(l)
used[l["person"]] += c
elif dec["decision"] in ("conflict", "blackout"):
s = suggest(team, current, l)
if s:
dec["suggestion"] = "%s to %s" % (s["start"].strftime("%a %d %b"), s["end"].strftime("%a %d %b"))
elif dec["decision"] == "blackout":
t = trimmed(team, current, l)
if t:
dec["suggestion"] = "shorten to %s - %s (%d workday(s), back before the blackout)" % (
t["start"].strftime("%a %d %b"), t["end"].strftime("%a %d %b"), len(leave_days(team, t)))
decisions.append(dec)
approved_peak = defaultdict(int)
for l in current:
approved_peak[l["person"]] += len(set(leave_days(team, l)) & team["peak"])
for dec in decisions:
l = dec["leave"]
p = team["people"][l["person"]]
if dec["decision"] != "approve" and p.get("worked_peak_last_year") and set(leave_days(team, l)) & team["peak"]:
lucky = [n for n, k in approved_peak.items() if k and not team["people"][n].get("worked_peak_last_year")]
add("medium", "LV05", "%s worked the peak last year but this request is %s while %s get peak days off; "
"consider swapping." % (l["person"], dec["decision"], ", ".join(sorted(lucky)) or "nobody"))
g_final = gaps(team, current)
allowance = []
for n, p in team["people"].items():
planned_req = sum(dec["charge"] for dec in decisions if dec["leave"]["person"] == n and dec["decision"] == "approve")
appr = used[n] - planned_req
left = p["allowance"] - p["taken"] - used[n] if p["allowance"] else None
allowance.append({"person": n, "allowance": p["allowance"], "taken_before": p["taken"], "approved": appr,
"recommended": planned_req, "left": left, "peak_days_off": approved_peak.get(n, 0)})
if p["allowance"] and p["taken"] + appr > p["allowance"] + 1e-9:
add("high", "LV02", "%s's approved leave exceeds the allowance by %g days (%g + %g taken of %g); "
"confirm carry-over or unpaid leave with HR." % (n, p["taken"] + appr - p["allowance"], p["taken"],
appr, p["allowance"]))
cm = p.get("carryover_max")
if left is not None and cm is not None and left > float(cm):
add("low", "LV06", "%s will still have %g days left (carry-over limit %g); %g day(s) are lost unless booked."
% (n, left, float(cm), left - float(cm)))
for role, k in sorted(team["coverage"].items()):
if role == "any":
continue
holders = [n for n, p in team["people"].items() if role in p["roles"]]
if len(holders) <= k:
add("medium", "LV03", "Only %d people can cover %s (minimum %d): any absence by %s breaks coverage; "
"cross-train a backup." % (len(holders), role, k, ", ".join(holders)))
elif len(holders) == k + 1:
add("low", "LV03", "%s has one spare person (%s for a minimum of %d); two absences at once break it."
% (role, ", ".join(holders), k))
for day, s in g_final.items():
if day not in g_approved:
add("high", "LV01", "%s still short after decisions: %s" % (day, s))
for dec in decisions:
if dec["decision"] in ("conflict", "blackout"):
l = dec["leave"]
add("medium", "LV04", "%s %s-%s: %s (%s)%s" % (
l["person"], l["start"].strftime("%d %b"), l["end"].strftime("%d %b"), dec["decision"], dec["reason"],
("; offer " + dec["suggestion"]) if dec["suggestion"] else "; no gap-free alternative within 15 days"))
elif dec["decision"] == "over allowance":
l = dec["leave"]
add("medium", "LV02", "%s %s-%s: %s; approve a shorter trip or agree unpaid days" % (
l["person"], l["start"].strftime("%d %b"), l["end"].strftime("%d %b"), dec["reason"]))
thin = []
off = off_map(team, current)
for day in team["days"]:
if not is_workday(team, day):
continue
working = [n for n, p in team["people"].items() if n not in off.get(day, set()) and person_works(team, p, day)]
if len(working) == need_for(team, day).get("any", 0):
thin.append(day)
if thin:
add("info", "LV07", "No slack on %d day(s) (%s): one sick day breaks coverage; line up a backup." % (
len(thin), ", ".join(x.strftime("%d %b") for x in thin)))
order = {"high": 0, "medium": 1, "low": 2, "info": 3}
findings.sort(key=lambda f: (order[f["severity"]], f["code"]))
return {"team": team.get("team", "Team"), "period": team["period"], "gaps_approved": g_approved,
"gaps_final": g_final, "decisions": decisions, "allowance": allowance, "findings": findings,
"current": current, "thin": thin}
# ----------------------------------------------------------------------------- report
def grid(team, leaves_final, all_leaves):
days = [x for x in team["days"] if x.weekday() < 5 or DAYS[x.weekday()] in team["workdays"]]
final_ids = {l["id"] for l in leaves_final}
cell = defaultdict(dict)
for l in all_leaves:
if l["status"] == "declined":
continue
for day in leave_days(team, l):
if l["id"] in final_ids:
c = TYPE_CODE[l["type"]] if l["status"] == "approved" else TYPE_CODE[l["type"]].lower()
else:
c = "x"
if l["half"]:
c = "h" if c != "x" else "x"
cell[l["person"]][day] = c
width = max([len(n) for n in team["people"]] + [len("cover " + r) for r in team["coverage"]]) + 2
months = " " * width + "".join(x.strftime("%b")[0] if i == 0 or x.month != days[i - 1].month
else " " for i, x in enumerate(days))
L = [months, " " * width + "".join(str(x.day // 10) for x in days),
" " * width + "".join(str(x.day % 10) for x in days),
" " * width + "".join(DAYS[x.weekday()][0].upper() for x in days)]
for n, p in team["people"].items():
row = []
for x in days:
if x in team["holidays"]:
row.append("H")
elif not person_works(team, p, x):
row.append("-")
else:
row.append(cell[n].get(x, "."))
L.append(n.ljust(width) + "".join(row))
off = off_map(team, leaves_final)
for role in sorted(team["coverage"]):
row = []
for x in days:
if not is_workday(team, x):
row.append(" ")
continue
s = shortfall(team, x, off.get(x, set()))
row.append("!" if role in s else "+")
L.append(("cover " + role).ljust(width) + "".join(row))
return L
def report(team, leaves, r):
L = ["# Leave coverage plan: %s (%s)" % (r["team"], r["period"].replace("..", " to ")), ""]
need = ", ".join("%s %d" % (k, v) for k, v in sorted(team["coverage"].items()))
L.append("Minimum cover per workday: %s. Holidays (closed): %s." % (
need, ", ".join(x.strftime("%d %b") for x in sorted(team["holidays"])) or "none"))
n_appr = sum(1 for dec in r["decisions"] if dec["decision"] == "approve")
verdict = ("approved leave already breaks coverage" if r["gaps_approved"] else
"over allowance on approved leave" if any(f["code"] == "LV02" and f["severity"] == "high" for f in r["findings"])
else "covered")
L += ["", "**Verdict: %s** - %d of %d open requests can be approved without a gap; %d need a decision." % (
verdict, n_appr, len(r["decisions"]), len(r["decisions"]) - n_appr), ""]
L += ["## Requests (in decision order: worked peak last year first, then first come first served)", "",
"| # | Person | Dates | Workdays | Type | Submitted | Decision | Why | Alternative |",
"|---|---|---|---|---|---|---|---|---|"]
for i, dec in enumerate(r["decisions"], 1):
l = dec["leave"]
L.append("| %d | %s | %s - %s%s | %s | %s | %s | %s | %s | %s |" % (
i, l["person"], l["start"].strftime("%a %d %b"), l["end"].strftime("%a %d %b"), " (half)" if l["half"] else "",
("%g" % (0.5 * dec["workdays"])) if l["half"] else dec["workdays"], l["type"], l["submitted"][:10],
dec["decision"], dec["reason"] or "-", dec["suggestion"] or "-"))
L += ["", "## Calendar (approved + recommended)", "", "```"] + grid(team, r["current"], leaves) + [
"```", "V/T/S/P/U/O approved vacation/training/sick/parental/unpaid/other, lowercase = request recommended "
"for approval, h = half day, x = request not approved, H = holiday, - = not a working day, "
"+ / ! = role covered / short.", ""]
L += ["## Allowance (days)", "", "| Person | Allowance | Taken before | Approved here | Recommended | Left | Peak days off |",
"|---|---|---|---|---|---|---|"]
for a in r["allowance"]:
L.append("| %s | %g | %g | %g | %g | %s | %d |" % (a["person"], a["allowance"], a["taken_before"], a["approved"],
a["recommended"], "-" if a["left"] is None else "%g" % a["left"],
a["peak_days_off"]))
L += ["", "## Findings", ""]
if not r["findings"]:
L.append("- none")
for f in r["findings"]:
L.append("- **%s %s** %s" % (f["severity"].upper(), f["code"], f["message"]))
return "\n".join(L) + "\n"
# ----------------------------------------------------------------------------- self-test
def self_test():
import os
import tempfile
cases = []
def ok(name, cond, detail=""):
cases.append((name, bool(cond), detail))
base = {"team": "T", "period": "2026-12-14..2026-12-25", "holidays": ["2026-12-25"], "peak_days": "2026-12-21..2026-12-24",
"coverage": {"lead": 1, "dev": 1, "any": 2},
"people": [{"name": "A", "roles": ["lead", "dev"], "allowance": 10, "taken": 0},
{"name": "B", "roles": ["dev"], "allowance": 10, "taken": 9},
{"name": "C", "roles": ["lead"], "allowance": 10, "taken": 0, "worked_peak_last_year": True},
{"name": "D", "roles": ["dev"], "days": ["mon", "tue"], "allowance": 5, "taken": 0}]}
t = load_team(base)
ok("weekends and holidays are not workdays", not is_workday(t, d("2026-12-19")) and not is_workday(t, d("2026-12-25")))
sf = shortfall(t, d("2026-12-16"), {"B", "C", "D"})
ok("one person fills one slot (A cannot be lead and dev)", sf.get("any") == 1 and sf.get("lead", 0) + sf.get("dev", 0) == 1, sf)
ok("matching finds A=dev, C=lead", shortfall(t, d("2026-12-16"), {"B", "D"}) == {})
ok("part-timer only counts on own days", shortfall(t, d("2026-12-17"), {"A", "B"}) == {"dev": 1, "any": 1}, shortfall(t, d("2026-12-17"), {"A", "B"}))
lv = {"id": 1, "person": "D", "start": d("2026-12-14"), "end": d("2026-12-20"), "type": "vacation",
"status": "requested", "submitted": "2026-11-01", "half": False, "notes": ""}
ok("leave days skip non-working days", len(leave_days(t, lv)) == 2)
ok("half day charges 0.5", charge(t, dict(lv, half=True)) == 1.0)
d0 = tempfile.mkdtemp()
csvp = os.path.join(d0, "l.csv")
with open(csvp, "w") as f:
f.write("person,start,end,type,status,submitted,half_day,notes\n"
"A,2026-12-21,2026-12-22,vacation,approved,2026-10-01,,\n"
"C,2026-12-21,2026-12-21,vacation,requested,2026-11-10,,\n"
"B,2026-12-14,2026-12-16,vacation,requested,2026-10-05,,\n"
"D,2026-12-15,2026-12-15,training,requested,2026-11-01,,\n")
leaves = load_leave(csvp, t)
r = plan(t, leaves)
by = {(dec["leave"]["person"]): dec for dec in r["decisions"]}
ok("C conflicts with A's approved leave (no lead)", by["C"]["decision"] == "conflict", by["C"])
ok("conflict gets an alternative date", by["C"]["suggestion"] is not None, by["C"])
ok("B over allowance (3 days, 1 left)", by["B"]["decision"] == "over allowance", by["B"])
ok("D training approved", by["D"]["decision"] == "approve", by["D"])
ok("worked-peak person decided first", r["decisions"][0]["leave"]["person"] == "C", [x["leave"]["person"] for x in r["decisions"]])
codes = {f["code"] for f in r["findings"]}
ok("LV03 single point of failure for lead", "LV03" in codes, codes)
ok("LV05 fairness flag for C", "LV05" in codes, codes)
ok("no gaps from approved leave", not r["gaps_approved"])
with open(csvp, "a") as f:
f.write("C,2026-12-21,2026-12-21,sick,approved,2026-12-01,,\n")
r2 = plan(t, load_leave(csvp, t))
ok("LV01 gap in approved leave", any(f["code"] == "LV01" and f["severity"] == "high" for f in r2["findings"]))
t2 = load_team(dict(base, blackout=[{"dates": "2026-12-16", "reason": "release"}]))
r3 = plan(t2, load_leave(csvp, t2))
ok("blackout precedes coverage", any(x["decision"] == "blackout" for x in r3["decisions"]) or
all(x["leave"]["person"] != "B" or x["decision"] == "over allowance" for x in r3["decisions"]))
lvb = dict(lv, person="A", start=d("2026-12-16"), end=d("2026-12-16"))
ok("blackout_hit detects overlap", blackout_hit(t2, lvb)[0] == [d("2026-12-16")])
try:
load_team(dict(base, coverage={"qa": 1}))
ok("unknown coverage role rejected", False)
except InputError:
ok("unknown coverage role rejected", True)
rep = report(t, leaves, r)
ok("report has grid and verdict", "Verdict" in rep and "cover lead" in rep)
for name, good, detail in cases:
print("%s %s%s" % ("PASS" if good else "FAIL", name, "" if good else " -> %r" % (detail,)))
passed = sum(1 for _, g, _ in cases if g)
print("%d/%d passed" % (passed, len(cases)))
return 0 if passed == len(cases) else 1
def to_json(r):
def conv(o):
if isinstance(o, dt.date):
return o.isoformat()
if isinstance(o, dict):
return {(k.isoformat() if isinstance(k, dt.date) else k): conv(v) for k, v in o.items()}
if isinstance(o, (list, tuple)):
return [conv(x) for x in o]
return o
return json.dumps(conv(r), indent=2)
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan team leave against minimum coverage.")
ap.add_argument("team", nargs="?")
ap.add_argument("leave", nargs="?")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not (a.team and a.leave):
ap.print_usage(sys.stderr)
return 2
try:
with open(a.team, encoding="utf-8") as f:
team = load_team(json.load(f))
leaves = load_leave(a.leave, team)
r = plan(team, leaves)
except (InputError, OSError, json.JSONDecodeError, KeyError) as e:
print("error: %s" % e, file=sys.stderr)
return 2
print(to_json(r) if a.json else report(team, leaves, r), end="\n" if a.json else "")
bad = r["gaps_approved"] or any(f["code"] == "LV02" and f["severity"] == "high" for f in r["findings"])
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
---
name: python-requirements-pin-and-conflict-checker
description: Checks pip requirements and constraints files offline before they break a build - unpinned and lower-bound-only packages, PEP 440 specifier conflicts across files and markers, git refs without a commit hash, editable and local paths, partial hash mode, pre-releases, --extra-index-url dependency confusion, http indexes, stdlib backports, dev tools in runtime files and pip freeze drift. Includes a stdlib Python checker with --self-test.
---
# Python Requirements Pin and Conflict Checker
You help Python developers and DevOps engineers make `pip install -r requirements.txt` boring: the same versions on every machine, no ResolutionImpossible at 2 a.m., no package quietly fetched from the wrong index. Most broken installs come from a few patterns that are visible in the files themselves: a constraint file that contradicts a requirement, `requests` with no pin, a git dependency on `@main`, one line with `--hash` that switches the whole file into hash mode, or an extra index that lets a public package shadow an internal one. You read every file pip would read, check the specifiers with real PEP 440 rules, and turn the result into a short fix list and a lock-file plan.
## Files in this skill
- `scripts/req_check.py` - parses requirements files the way pip does (`-r` includes, `-c` constraints, line continuations, comments, extras, PEP 508 markers, `--hash`, index options, VCS/URL/editable lines), evaluates PEP 440 specifiers including `~=`, wildcards and pre-releases, searches for a version that satisfies every specifier per marker environment, compares with `pip freeze` output, and reports findings PR01-PR14 (Python 3.8+ standard library only, no network)
- `references/specifiers-and-markers.md` - PEP 440 operators, pre-release rules, constraints vs requirements, marker evaluation, and why the checker cannot see transitive dependencies
- `references/pinning-strategy.md` - apps vs libraries, the requirements.in + lock workflow, hashes, indexes and supply-chain hygiene, and what each finding code means
- `templates/requirements-review.md` - the review to hand back
- `examples/requirements.txt` - a Django invoice service with a typo, a git branch dependency, an extra index, a marker-split numpy, a duplicate Django, enum34 and pytest in runtime
- `examples/constraints.txt` - the platform team's constraints file that contradicts two of those requirements
- `examples/req-report.md` - the real checker output on those files
## Workflow
### 1. Collect the inputs
Ask for every file pip reads for the target: the main requirements file, anything it includes with `-r` or `-c`, plus `pip.conf` / `PIP_INDEX_URL` / `PIP_EXTRA_INDEX_URL` settings from CI and Dockerfiles (index options outside the file are invisible to the checker; add them by hand). Ask whether the project is an **app** (deployed service, notebook image, CLI shipped as a container: pins expected) or a **library** (installed by other projects: ranges expected), which Python versions and platforms it targets, and for a `pip freeze` from the environment that misbehaves.
### 2. Run the checker
```
python3 scripts/req_check.py examples/requirements.txt
python3 scripts/req_check.py requirements.txt --python 3.12 --platform linux
python3 scripts/req_check.py requirements.txt requirements-dev.txt --installed freeze.txt
python3 scripts/req_check.py requirements.txt --mode lib --json
python3 scripts/req_check.py --self-test
```
Exit 0 = no high findings, 1 = high findings, 2 = a file could not be read. Without `--python`, each marker combination is checked as its own environment; with it, lines whose markers do not apply are dropped. Without a shell, go line by line with `references/specifiers-and-markers.md` and test each package's combined specifiers against a concrete version.
### 3. Read the result
- Install blockers first: PR03 (no version satisfies the combined specifiers), PR07 (partial hash mode), PR13 (lines pip refuses). These fail before anything installs.
- Supply chain: PR09 extra index, PR10 http or trusted hosts, PR05 git refs that can move, PR11 bogus or dangerous packages (`sklearn`, `enum34`, `pycrypto`).
- Reproducibility: PR01 unpinned and PR02 open-ended ranges in apps; in libraries, PR02 flags exact pins that would force versions on users.
- Drift: PR12 shows where the running environment differs from the files.
- Tidy-ups: PR04 duplicates, PR06 editable/local paths, PR08 pre-releases, PR14 dev tools in runtime files.
### 4. Fix and lock
Resolve each conflict by deciding which side is right (often the constraint is stale; check the changelog of the newer version), then move intent-level ranges to `requirements.in` and generate the pinned, hashed lock with pip-tools or uv (`references/pinning-strategy.md`). Re-run the checker on the lock: it should come back with no high or medium findings. Never "fix" a conflict by deleting the constraint without asking who owns it.
### 5. Deliver
Fill `templates/requirements-review.md`: verdict, blockers with exact replacement lines, supply-chain changes, the lock workflow and CI commands, and an owner for each change.
## Example result
On `examples/requirements.txt` the checker finds 6 high findings. Two conflicts make pip fail: `urllib3>=2.2` against the platform constraint `urllib3<2`, and `numpy>=2.1` against the constraint `numpy==1.26.4` for Python 3.9+ (the Python < 3.9 branch is fine). `pandas=2.2.3` uses a single `=` and is refused. `pdf-stamp` is installed editable from `@main`, the internal index is added with `--extra-index-url` (dependency confusion risk for `acme-billing-core`), and `enum34` breaks the standard library `enum` on modern Python. Medium and low: open-ended `whitenoise` and `urllib3`, the editable install, Django declared twice as `Django` and `django`, and `pytest` in the runtime file.
## Rules
- Read every file pip reads, including constraints and includes; a single file in isolation hides conflicts.
- Use PEP 440 semantics, not string comparison (`2.10` > `2.9`, `~=2.1` allows 2.9 but not 3.0, `<2.0` excludes `2.0rc1`).
- Say clearly that the checker sees only declared packages: transitive conflicts and CVEs need the resolver (`pip install --dry-run`, `pip-compile`, `uv pip compile`) and `pip-audit`.
- Prefer a lock file generated by a tool over hand pins; never invent version numbers - take them from the current environment, the lock, or the package changelog.
- Treat index and VCS findings as security issues, not style.
FILE:references/specifiers-and-markers.md
# Specifiers, constraints and markers
## PEP 440 operators
| Specifier | Means | Example matches | Does not match |
|---|---|---|---|
| `==2.32.3` | exactly this version (trailing zeros ignored: `==2.0` = `==2.0.0`) | 2.32.3 | 2.32.4 |
| `==2.32.*` | any 2.32.x | 2.32.0, 2.32.9 | 2.33.0 |
| `~=2.31` | `>=2.31, ==2.*` | 2.31, 2.40 | 3.0 |
| `~=2.31.0` | `>=2.31.0, ==2.31.*` | 2.31.5 | 2.32.0 |
| `>=4.2,<5.0` | range | 4.2.16 | 5.0 |
| `!=3.1.2` | anything but | 3.1.3 | 3.1.2 |
| `===foo` | arbitrary string equality (avoid) | foo | |
- Order: `1.0.dev1 < 1.0a1 < 1.0b1 < 1.0rc1 < 1.0 < 1.0.post1`; `1.10 > 1.9` (numeric, not text).
- `~=2` is invalid (needs at least two release parts); `pkg=1.0` is invalid (single `=`).
- Pre-releases are only chosen if a specifier names one (`>=2.0rc1`), you pass `--pre`, or nothing else matches.
- `<2.0` excludes 2.0 pre-releases; `>1.4` excludes 1.4 post-releases.
- Local versions (`1.0+cpu`) only match `==1.0+cpu` or ranges; `==1.0` matches `1.0+cpu` too.
## Requirements vs constraints
- A requirement (`pkg>=1`) asks pip to install the package.
- A constraint (`-c constraints.txt`, `PIP_CONSTRAINT`) only restricts versions if something requires the package; it never installs anything. Constraints cannot have extras or be editable.
- pip combines all specifiers for a name (after PEP 503 normalisation: `Django`, `django`, `DJANGO` and `foo_bar` / `foo-bar` / `foo.bar` are the same). If nothing satisfies all of them, the resolver stops with ResolutionImpossible.
## Markers (PEP 508)
- `numpy<2 ; python_version < "3.9"` - the line applies only where the marker is true.
- Variables: `python_version` ("3.12"), `python_full_version`, `sys_platform` (linux, win32, darwin), `platform_system` (Linux, Windows, Darwin), `os_name` (posix, nt), `platform_machine`, `implementation_name`, `extra`.
- Two lines for one package with opposite markers are fine; two lines with the same (or no) marker are duplicates.
- The checker evaluates markers when you pass `--python` / `--platform`; otherwise it checks the unmarked lines plus each marker set as a separate environment.
## What an offline file check cannot see
- Transitive dependencies: `requests==2.32.3` needs `urllib3>=1.21.1,<3`; a conflict between what your packages need is only visible to the resolver. Run `pip install --dry-run --report report.json -r requirements.txt` or compile a lock.
- Wheels: a pinned version may have no wheel for your Python/platform (e.g. a new Python release) and fall back to a source build.
- Yanked releases and CVEs: use `pip-audit` and the index.
FILE:references/pinning-strategy.md
# Pinning strategy and supply-chain hygiene
## Apps vs libraries
| | App (service, job, notebook image) | Library (installed by others) |
|---|---|---|
| Direct deps | ranges in `requirements.in` / pyproject | ranges in pyproject `dependencies` |
| What gets installed | a lock: every package `==` + `--hash` | whatever the user's resolver picks |
| Exact pins | required (in the lock) | avoid: they force versions on every user |
| Upper bounds | let the lock hold them | only for known breakage (`<3` when 3.0 broke you) |
## The lock workflow (apps)
1. `requirements.in`: what you import, with intent-level ranges (`Django>=4.2,<5`).
2. Compile: `pip-compile --generate-hashes -o requirements.txt requirements.in` or `uv pip compile --generate-hashes requirements.in -o requirements.txt`. Add `-c constraints.txt` if the platform team ships constraints.
3. Install: `pip install --require-hashes --no-deps -r requirements.txt` (the lock already contains everything).
4. Update on purpose: `pip-compile --upgrade-package django`, review the diff, run tests. Automate with Dependabot/Renovate.
5. Dev tools go to `requirements-dev.in` compiled with `-c requirements.txt` so versions agree.
## Hashes
- One `--hash` anywhere turns on hash-checking mode for the whole install: every package (including transitive ones) needs `==` and a hash, or pip refuses. That is why partial hashes (PR07) break installs.
- Generate hashes with a tool, never by hand; include every platform's wheel hash if you build on several.
## Indexes
- `--extra-index-url` makes pip look at both indexes and take the highest version: anyone can upload `acme-billing-core==99.0` to PyPI and win (dependency confusion). Use one `--index-url` that points to a proxy (Artifactory, Nexus, devpi, CodeArtifact, Azure Artifacts) serving PyPI plus internal packages, and reserve internal names on PyPI.
- https only; `--trusted-host` turns off certificate checks.
- VCS dependencies: pin a full commit (`@3f2a...` 40 hex); branches and tags move. Better: publish a wheel to the internal index.
## Finding codes (req_check.py)
| Code | Severity | Meaning |
|---|---|---|
| PR01 | high (app) / info (lib) | no version specifier and no pin elsewhere |
| PR02 | medium | app: only `>=`/`>`; lib: exact `==` pin |
| PR03 | high | no version satisfies the combined specifiers (per marker environment) |
| PR04 | low / info | declared more than once, or spelt differently |
| PR05 | high / medium | VCS ref without commit hash / archive URL without hash |
| PR06 | medium | editable or local path |
| PR07 | high | partial hash mode, or hashed line without `==` |
| PR08 | medium | pre-release pin or `--pre` |
| PR09 | high | `--extra-index-url` |
| PR10 | high / medium | http index or URL / `--trusted-host` |
| PR11 | varies | bogus, unmaintained or backport package (sklearn, pycrypto, enum34, typing, nose ...) |
| PR12 | high / medium | installed version does not satisfy the files / package missing |
| PR13 | high | line pip refuses (bad specifier, missing include) |
| PR14 | low | test or dev tool in a runtime file |
FILE:templates/requirements-review.md
# Requirements review: <project>
**Files:** <requirements.txt, constraints.txt (-c)> **Kind:** <app / library> **Python:** <3.12, linux>
**Verdict:** <install fails / installs but not reproducible / reproducible>
## Blockers (install fails)
| # | Where | Problem | Replace with |
|---|---|---|---|
| 1 | <constraints.txt:3 + requirements.txt:24> | <urllib3 <2 vs >=2.2> | <agree on urllib3>=2.2,<3 with platform team; update PLAT-412> |
| 2 | <requirements.txt:20> | <pandas=2.2.3> | <pandas==2.2.3> |
## Supply chain
| # | Where | Problem | Fix |
|---|---|---|---|
| 1 | <requirements.txt:5> | <--extra-index-url internal> | <single --index-url to the proxy; reserve acme-* on PyPI> |
| 2 | <requirements.txt:28> | <pdf-stamp @main, editable> | <publish a wheel or pin @<commit>> |
## Reproducibility and tidy-ups
- <unpinned / open ranges: ...>
- <duplicates, backports, dev tools: ...>
## Lock plan
```
<requirements.in contents (intent ranges)>
pip-compile --generate-hashes -c constraints.txt -o requirements.txt requirements.in
pip install --require-hashes --no-deps -r requirements.txt
```
## Not covered by this check
<transitive conflicts: run pip install --dry-run / pip-compile; CVEs: pip-audit; wheel availability for Python X.Y>
## Owners and dates
| Change | Owner | By |
|---|---|---|
| <...> | <...> | <...> |
FILE:examples/requirements.txt
# invoice-service runtime dependencies (deployed as a Docker image, Python 3.12)
# org-wide pins from the platform team:
-c constraints.txt
--index-url https://pypi.org/simple
--extra-index-url https://pkgs.acme.internal/simple
# web
Django>=4.2,<5.0
djangorestframework~=3.15
gunicorn==23.0.0
whitenoise>=6.7
# data and storage
psycopg[binary]==3.2.3
redis>=5.0
celery[redis]==5.4.0
boto3
numpy<2 ; python_version < "3.9"
numpy>=2.1 ; python_version >= "3.9"
pandas=2.2.3
# http
requests==2.32.3
urllib3>=2.2
# pdf invoices
weasyprint==63.0
-e git+https://github.com/acme/pdf-stamp.git@main#egg=pdf-stamp
acme-billing-core==1.8.2
# misc
python-dateutil==2.9.0.post0
sentry-sdk>=2.0
django==4.2.16
enum34==1.1.10
pytest==8.3.3
FILE:examples/constraints.txt
# Platform team constraints, applied to every service with -c
# (pins only; a constraint never installs anything by itself)
urllib3<2 # legacy proxy TLS issue, see PLAT-412
numpy==1.26.4 # matches the shared analytics image
redis==5.0.8
boto3==1.35.36
sentry-sdk==2.17.0
celery==5.4.0
cryptography>=43.0.1 # CVE floor
FILE:examples/req-report.md
# Requirements check: examples/requirements.txt
Mode: app. Packages: 19 (14 pinned with ==). Findings: 6 high, 3 medium, 2 low, 0 info.
**Verdict: install will fail or is unsafe - fix the high findings first**
## Packages
| Package | Specifiers (all files) | Marker | Declared in | Status |
|---|---|---|---|---|
| urllib3 | `<2,>=2.2` | - | examples/constraints.txt:3 (constraint), examples/requirements.txt:24 | conflict |
| numpy | `<2,==1.26.4,>=2.1` | `python_version < "3.9"; python_version >= "3.9"` | examples/constraints.txt:4 (constraint), examples/requirements.txt:18, examples/requirements.txt:19 | conflict |
| redis | `==5.0.8,>=5.0` | - | examples/constraints.txt:5 (constraint), examples/requirements.txt:15 | ok |
| boto3 | `==1.35.36` | - | examples/constraints.txt:6 (constraint), examples/requirements.txt:17 | ok |
| sentry-sdk | `==2.17.0,>=2.0` | - | examples/constraints.txt:7 (constraint), examples/requirements.txt:33 | ok |
| celery | `==5.4.0` | - | examples/constraints.txt:8 (constraint), examples/requirements.txt:16 | ok |
| cryptography | `>=43.0.1` | - | examples/constraints.txt:9 (constraint) | ok |
| Django | `<5.0,==4.2.16,>=4.2` | - | examples/requirements.txt:8, examples/requirements.txt:34 | ok |
| djangorestframework | `~=3.15` | - | examples/requirements.txt:9 | ok |
| gunicorn | `==23.0.0` | - | examples/requirements.txt:10 | ok |
| whitenoise | `>=6.7` | - | examples/requirements.txt:11 | ok |
| psycopg | `==3.2.3` | - | examples/requirements.txt:14 | ok |
| requests | `==2.32.3` | - | examples/requirements.txt:23 | ok |
| weasyprint | `==63.0` | - | examples/requirements.txt:27 | ok |
| pdf-stamp | `url` | - | examples/requirements.txt:28 | ok |
| acme-billing-core | `==1.8.2` | - | examples/requirements.txt:29 | ok |
| python-dateutil | `==2.9.0.post0` | - | examples/requirements.txt:32 | ok |
| enum34 | `==1.1.10` | - | examples/requirements.txt:35 | ok |
| pytest | `==8.3.3` | - | examples/requirements.txt:36 | ok |
## Findings
- **HIGH PR03** `examples/constraints.txt:3, examples/requirements.txt:24` urllib3: no version satisfies <2 (examples/constraints.txt:3), >=2.2 (examples/requirements.txt:24) - pip will fail with ResolutionImpossible
- **HIGH PR03** `examples/constraints.txt:4, examples/requirements.txt:19` numpy: no version satisfies ==1.26.4 (examples/constraints.txt:4), >=2.1 (examples/requirements.txt:19) when python_version >= "3.9" - pip will fail with ResolutionImpossible
- **HIGH PR05** `examples/requirements.txt:28` pdf-stamp comes from git+https://github.com/acme/pdf-stamp.git@main#egg=pdf-stamp without a full commit hash; the branch or tag can move under you - pin @<40-char commit>
- **HIGH PR09** `examples/requirements.txt:5` --extra-index-url https://pkgs.acme.internal/simple: pip picks the highest version from ANY index, so a public package with the same name as an internal one wins (dependency confusion); use a single --index-url proxy that serves both, or pin with --hash
- **HIGH PR11** `examples/requirements.txt:35` enum34 breaks the stdlib enum on Python 3.6+ (e.g. AttributeError in re); remove it
- **HIGH PR13** `examples/requirements.txt:20` pandas: single '=' in '=2.2.3' (use ==) - pip will refuse this line
- **MEDIUM PR02** `examples/requirements.txt:11` whitenoise >=6.7 has no upper bound and no pin; a future major release installs automatically
- **MEDIUM PR02** `examples/requirements.txt:24` urllib3 >=2.2 has no upper bound and no pin; a future major release installs automatically
- **MEDIUM PR06** `examples/requirements.txt:28` pdf-stamp is an editable install; it will not exist on a clean build machine - publish it or build a wheel
- **LOW PR04** `examples/requirements.txt:8, examples/requirements.txt:34` Django is declared 2 times (spelt Django, django); keep one line
- **LOW PR14** `examples/requirements.txt:36` pytest is a development/test tool in a runtime requirements file; move it to requirements-dev.txt so production images stay small
## Next steps
1. Fix every PR03/PR07/PR13 line - these stop `pip install` outright.
2. Keep loose, intent-level ranges in `requirements.in` (or pyproject) and generate a fully pinned, hashed lock with `pip-compile --generate-hashes` or `uv pip compile --generate-hashes`; install with `pip install --require-hashes -r requirements.txt`.
3. Use one `--index-url` (a proxy that mirrors PyPI and hosts internal packages), https only.
4. Run `pip-audit -r requirements.txt` (online) for known vulnerabilities - this checker is offline and does not know CVEs.
FILE:scripts/req_check.py
#!/usr/bin/env python3
"""Python requirements pin and conflict checker (Python 3.8+ standard library only, offline).
Reads pip requirements files (following -r includes and -c constraints), and reports what makes
installs non-reproducible, unsatisfiable or risky:
PR01 unpinned requirement PR08 pre-release pin or --pre
PR02 lower bound only (>=, >) PR09 --extra-index-url (dependency confusion)
PR03 conflicting specifiers PR10 plain http:// index/URL, --trusted-host
PR04 duplicate / differently spelt PR11 deprecated, renamed or stdlib-backport package
PR05 VCS/URL without immutable ref PR12 installed version (pip freeze) does not match
PR06 editable or local path PR13 line pip cannot parse
PR07 incomplete hash-checking mode PR14 dev/test tool in runtime requirements
Specifiers follow PEP 440 (==, !=, <=, >=, <, >, ~=, ===, ==X.*), names PEP 503 normalisation,
markers PEP 508 (evaluated with --python / --platform, otherwise each marker set is checked as its
own environment).
Usage:
python3 req_check.py requirements.txt [more.txt ...] [--mode app|lib] [--installed freeze.txt]
[--python 3.11] [--platform linux] [--json]
python3 req_check.py --self-test
Exit codes: 0 = no high findings, 1 = high findings, 2 = file missing or unreadable.
"""
import argparse
import json
import os
import re
import sys
from collections import OrderedDict, defaultdict
# ----------------------------------------------------------------------------- versions (PEP 440)
VERSION_RE = re.compile(r"""^\s*v?(?:(?P<epoch>\d+)!)?(?P<release>\d+(?:\.\d+)*)
(?:[-_.]?(?P<pre_l>a|alpha|b|beta|c|rc|pre|preview)[-_.]?(?P<pre_n>\d*))?
(?:-(?P<post_n1>\d+)|[-_.]?(?P<post_l>post|rev|r)[-_.]?(?P<post_n2>\d*))?
(?:[-_.]?(?P<dev_l>dev)[-_.]?(?P<dev_n>\d*))?
(?:\+(?P<local>[a-z0-9]+(?:[-_.][a-z0-9]+)*))?\s*$""", re.I | re.X)
PRE_RANK = {"a": 0, "alpha": 0, "b": 1, "beta": 1, "c": 2, "rc": 2, "pre": 2, "preview": 2}
INF = float("inf")
class Version:
def __init__(self, s):
m = VERSION_RE.match(s)
if not m:
raise ValueError("invalid version %r" % s)
self.text = s.strip()
self.epoch = int(m.group("epoch") or 0)
self.release = tuple(int(x) for x in m.group("release").split("."))
self.pre = (PRE_RANK[m.group("pre_l").lower()], int(m.group("pre_n") or 0)) if m.group("pre_l") else None
post = m.group("post_n1") or (m.group("post_n2") or "0" if m.group("post_l") else None)
self.post = int(post) if post is not None else None
self.dev = int(m.group("dev_n") or 0) if m.group("dev_l") else None
self.local = m.group("local")
rel = list(self.release)
while len(rel) > 1 and rel[-1] == 0:
rel.pop()
if self.pre is None and self.post is None and self.dev is not None:
pre = (-1, 0)
else:
pre = self.pre if self.pre is not None else (3, 0)
self.key = (self.epoch, tuple(rel), pre, -1 if self.post is None else self.post,
INF if self.dev is None else self.dev)
@property
def is_prerelease(self):
return self.pre is not None or self.dev is not None
def __lt__(self, o):
return self.key < o.key
def __eq__(self, o):
return self.key == o.key
def __hash__(self):
return hash(self.key)
def __repr__(self):
return self.text
SPEC_RE = re.compile(r"^\s*(===|==|!=|<=|>=|~=|<|>)\s*([^\s,;]+)\s*$")
class Spec:
def __init__(self, text):
m = SPEC_RE.match(text)
if not m:
raise ValueError("invalid specifier %r" % text.strip())
self.op, self.ver_text = m.group(1), m.group(2)
self.wild = self.ver_text.endswith(".*")
if self.op == "===":
self.version = None
return
if self.wild and self.op not in ("==", "!="):
raise ValueError("wildcard only allowed with == or != (%r)" % text.strip())
self.version = Version(self.ver_text[:-2] if self.wild else self.ver_text)
if self.op == "~=" and len(self.version.release) < 2:
raise ValueError("~= needs at least two release parts, e.g. ~=2.1 (%r)" % text.strip())
def __repr__(self):
return self.op + self.ver_text
def contains(self, v):
op, s = self.op, self.version
if op == "===":
return v.text == self.ver_text
if self.wild:
n = len(s.release)
rel = tuple(list(v.release) + [0] * max(0, n - len(v.release)))[:n]
hit = v.epoch == s.epoch and rel == s.release
return hit if op == "==" else not hit
vk = v.key
if op == "==":
return vk == s.key and (s.local is None or v.local == s.local)
if op == "!=":
return vk != s.key
if op == ">=":
return vk >= s.key
if op == "<=":
return vk <= s.key
if op == ">":
return vk > s.key and not (v.post is not None and v.release == s.release and s.post is None)
if op == "<":
return vk < s.key and not (v.is_prerelease and not s.is_prerelease and
Version(".".join(map(str, v.release))).key == s.key)
if op == "~=":
prefix = s.release[:-1]
rel = tuple(list(v.release) + [0] * len(s.release))[:len(prefix)]
return vk >= s.key and rel == prefix and v.epoch == s.epoch
return False
def is_prerelease_spec(self):
return self.version is not None and self.version.is_prerelease
def satisfiable(specs):
"""Return an example final version that satisfies all specs, or None."""
if not specs:
return "any"
for s in specs:
if s.op == "===":
try:
v = Version(s.ver_text)
except ValueError:
return s.ver_text
return v.text if all(x.contains(v) for x in specs) else None
cands = set()
for s in specs:
r = list(s.version.release)
e = "%d!" % s.version.epoch if s.version.epoch else ""
base = ".".join(map(str, r))
cands.update([s.version.text, e + base, e + base + ".0.0.1", e + base + ".post1"])
if s.wild or s.op == "~=":
pre = r if s.wild else r[:-1]
cands.update([e + ".".join(map(str, pre + [0])), e + ".".join(map(str, pre + [999]))])
nxt = pre[:-1] + [pre[-1] + 1]
cands.add(e + ".".join(map(str, nxt)))
for i in range(len(r)):
if r[i] > 0:
lower = r[:i] + [r[i] - 1] + [999]
cands.add(e + ".".join(map(str, lower)))
up = r[:i] + [r[i] + 1]
cands.add(e + ".".join(map(str, up)))
cands.update(["0", "0.0.1", "99999"])
allow_pre = any(s.is_prerelease_spec() for s in specs)
found = []
for c in cands:
try:
v = Version(c)
except ValueError:
continue
if v.is_prerelease and not allow_pre:
continue
if all(s.contains(v) for s in specs):
found.append(v)
return min(found).text if found else None
# ----------------------------------------------------------------------------- markers (PEP 508, subset)
MARKER_TOKEN = re.compile(r"\s*(\(|\)|===|==|!=|<=|>=|~=|<|>|not\s+in\b|in\b|and\b|or\b|'[^']*'|\"[^\"]*\"|[A-Za-z_][A-Za-z0-9_.]*)")
def eval_marker(text, env):
toks, pos = [], 0
while pos < len(text):
m = MARKER_TOKEN.match(text, pos)
if not m:
if text[pos:].strip() == "":
break
raise ValueError("cannot parse marker %r" % text)
toks.append(re.sub(r"\s+", " ", m.group(1)))
pos = m.end()
i = [0]
def value():
t = toks[i[0]]
i[0] += 1
if t[0] in "'\"":
return t[1:-1], False
if t not in env:
raise ValueError("unknown marker variable %r" % t)
return env[t], t.startswith("python") or t.endswith("_version")
def comparison():
if toks[i[0]] == "(":
i[0] += 1
r = disj()
i[0] += 1
return r
left, lv = value()
op = toks[i[0]]
i[0] += 1
right, rv = value()
if op in ("in", "not in"):
r = left in right
return r if op == "in" else not r
if lv or rv:
try:
return Spec(op + right).contains(Version(left))
except ValueError:
pass
return {"==": left == right, "!=": left != right, "<": left < right, ">": left > right,
"<=": left <= right, ">=": left >= right}.get(op, False)
def conj():
r = comparison()
while i[0] < len(toks) and toks[i[0]] == "and":
i[0] += 1
r = comparison() and r
return r
def disj():
r = conj()
while i[0] < len(toks) and toks[i[0]] == "or":
i[0] += 1
r = conj() or r
return r
return disj()
def marker_env(python, platform):
plat = (platform or "linux").lower()
sysplat = {"linux": "linux", "windows": "win32", "win32": "win32", "macos": "darwin", "darwin": "darwin"}.get(plat, plat)
system = {"linux": "Linux", "win32": "Windows", "darwin": "Darwin"}.get(sysplat, plat)
pv = ".".join(python.split(".")[:2])
return {"python_version": pv, "python_full_version": python if python.count(".") == 2 else python + ".0",
"sys_platform": sysplat, "platform_system": system, "os_name": "nt" if sysplat == "win32" else "posix",
"platform_machine": "x86_64", "implementation_name": "cpython", "platform_python_implementation": "CPython",
"extra": ""}
# ----------------------------------------------------------------------------- parsing
NAME_RE = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)\s*(\[[^\]]*\])?\s*(.*)$")
HEX40 = re.compile(r"@[0-9a-f]{40}(?:#|$)")
DEPRECATED = {
"sklearn": ("high", "is a dummy package that now fails to install; use scikit-learn"),
"bs4": ("low", "is a convenience alias; depend on beautifulsoup4 directly"),
"pycrypto": ("high", "is unmaintained since 2013 with known vulnerabilities; use pycryptodome or cryptography"),
"nose": ("medium", "is unmaintained and breaks on Python 3.10+; use pytest"),
"mock": ("low", "is a backport; on Python 3 use unittest.mock"),
"argparse": ("medium", "is in the standard library since 2.7/3.2; remove the backport"),
"typing": ("medium", "is a backport for Python < 3.5 and can shadow the stdlib module; remove it"),
"dataclasses": ("medium", "is a backport for Python 3.6 only; remove it or add ; python_version < \"3.7\""),
"enum34": ("high", "breaks the stdlib enum on Python 3.6+ (e.g. AttributeError in re); remove it"),
"futures": ("medium", "is a Python 2 backport of concurrent.futures; remove it"),
"pathlib": ("medium", "is an old backport that shadows the stdlib pathlib; remove it"),
"distribute": ("medium", "was merged back into setuptools in 2013; use setuptools"),
"pkg-resources": ("high", "is a bogus package generated by a broken Ubuntu pip freeze; remove the line"),
"python-dateutils": ("medium", "is a typo-like name; you probably meant python-dateutil"),
}
DEV_TOOLS = {"pytest", "pytest-cov", "pytest-django", "pytest-mock", "black", "flake8", "pylint", "mypy", "ruff",
"isort", "ipython", "ipdb", "pre-commit", "coverage", "tox", "nox", "factory-boy", "faker", "debugpy",
"django-debug-toolbar", "jupyter", "notebook"}
def show(path):
rel = os.path.relpath(path)
return path if rel.startswith("..") else rel
def norm(name):
return re.sub(r"[-_.]+", "-", name).lower()
class Req:
def __init__(self, **kw):
self.__dict__.update(kw)
def logical_lines(text):
buf, start = "", None
for n, raw in enumerate(text.splitlines(), 1):
line = raw
if start is None:
start = n
if line.rstrip().endswith("\\"):
buf += line.rstrip()[:-1] + " "
continue
buf += line
yield start, buf
buf, start = "", None
if buf:
yield start, buf
def strip_comment(line):
if line.lstrip().startswith("#"):
return ""
return re.split(r"\s+#", line, maxsplit=1)[0].strip()
def parse_files(paths, findings):
reqs, opts, seen = [], defaultdict(list), set()
def add(sev, code, src, msg):
findings.append({"severity": sev, "code": code, "where": src, "message": msg})
def parse(path, constraint):
real = os.path.abspath(path)
if real in seen:
return
seen.add(real)
with open(path, encoding="utf-8") as f:
text = f.read()
base = os.path.dirname(path)
shown = show(path)
for n, line in logical_lines(text):
src = "%s:%d" % (shown, n)
line = strip_comment(line)
if not line:
continue
m = re.match(r"^(-r|--requirement|-c|--constraint)\s*=?\s*(\S+)$", line)
if m:
inc = os.path.join(base, m.group(2))
if not os.path.exists(inc):
add("high", "PR13", src, "included file %s does not exist; pip will fail" % m.group(2))
else:
parse(inc, constraint or m.group(1) in ("-c", "--constraint"))
continue
m = re.match(r"^(-i|--index-url|--extra-index-url|-f|--find-links|--trusted-host)\s*=?\s*(\S+)$", line)
if m:
opt = {"-i": "--index-url", "-f": "--find-links"}.get(m.group(1), m.group(1))
opts[opt].append((src, m.group(2)))
continue
if line in ("--pre", "--require-hashes") or line.startswith(("--no-binary", "--only-binary", "--prefer-binary")):
opts[line.split()[0].split("=")[0]].append((src, line))
continue
editable = False
m = re.match(r"^(-e|--editable)\s+(\S+)(.*)$", line)
if m:
editable, line = True, m.group(2) + m.group(3)
hashes = re.findall(r"--hash[=\s]+(\S+)", line)
line = re.sub(r"--hash[=\s]+\S+", "", line).strip()
if line.startswith("-"):
add("medium", "PR13", src, "unknown option %r ignored by this checker" % line.split()[0])
continue
marker = ""
if ";" in line:
line, marker = [x.strip() for x in line.split(";", 1)]
url, name, extras, specs, bad = None, None, "", [], None
if re.match(r"^(git|hg|svn|bzr)\+|^https?://|^file:|^\.{0,2}/|.*\.(whl|zip|tar\.gz)$", line):
url = line
egg = re.search(r"[#&]egg=([A-Za-z0-9._-]+)", line)
name = egg.group(1) if egg else os.path.basename(line.split("#")[0].rstrip("/")).split("@")[0]
name = re.sub(r"(-\d.*)?(\.git|\.whl|\.zip|\.tar\.gz)$", "", name) or line
else:
m = NAME_RE.match(line)
if not m:
add("high", "PR13", src, "pip cannot parse %r" % line)
continue
name, extras, rest = m.group(1), m.group(2) or "", m.group(3).strip()
if rest.startswith("@"):
url = rest[1:].strip()
elif rest:
rest = rest.strip("()")
for part in [p for p in rest.split(",") if p.strip()]:
p = part.strip()
if re.match(r"^=[^=]", p):
bad = "single '=' in %r (use ==)" % p
break
try:
specs.append(Spec(p))
except ValueError as e:
bad = str(e)
break
if bad:
add("high", "PR13", src, "%s: %s - pip will refuse this line" % (name, bad))
continue
reqs.append(Req(name=name, key=norm(name), extras=extras, specs=specs, marker=marker, url=url,
editable=editable, hashes=hashes, constraint=constraint, src=src))
for p in paths:
parse(p, False)
return reqs, opts
# ----------------------------------------------------------------------------- checks
def check(paths, mode="app", installed=None, python=None, platform=None):
findings = []
reqs, opts = parse_files(paths, findings)
def add(sev, code, src, msg):
findings.append({"severity": sev, "code": code, "where": src, "message": msg})
env = marker_env(python, platform) if python else None
active = []
for r in reqs:
if r.marker and env is not None:
try:
if not eval_marker(r.marker, env):
continue
except (ValueError, IndexError):
add("medium", "PR13", r.src, "cannot evaluate marker %r" % r.marker)
active.append(r)
by = OrderedDict()
for r in active:
by.setdefault(r.key, []).append(r)
# index / transport options
for src, url in opts.get("--extra-index-url", []):
add("high", "PR09", src, "--extra-index-url %s: pip picks the highest version from ANY index, so a public "
"package with the same name as an internal one wins (dependency confusion); use a single --index-url "
"proxy that serves both, or pin with --hash" % url)
for opt in ("--index-url", "--extra-index-url", "--find-links"):
for src, url in opts.get(opt, []):
if url.startswith("http://"):
add("high", "PR10", src, "%s uses plain http (%s); packages can be swapped in transit - use https" % (opt, url))
for src, host in opts.get("--trusted-host", []):
add("medium", "PR10", src, "--trusted-host %s disables TLS verification for that host" % host)
for src, _ in opts.get("--pre", []):
add("medium", "PR08", src, "--pre lets every package resolve to pre-releases; pin the one pre-release you need instead")
hashed = [r for r in active if r.hashes and not r.constraint]
hash_mode = bool(hashed) or bool(opts.get("--require-hashes"))
pinned_elsewhere = {}
for key, rs in by.items():
for r in rs:
if any(s.op in ("==", "===") and not s.wild for s in r.specs):
pinned_elsewhere[key] = r.src
packages = []
for key, rs in by.items():
decl = [r for r in rs if not r.constraint]
all_specs = [s for r in rs for s in r.specs]
groups = defaultdict(list)
for r in rs:
groups[r.marker].append(r)
unmarked = groups.pop("", [])
envs = [("", unmarked)] if not groups else [(m, unmarked + g) for m, g in groups.items()]
status, example = "ok", None
for m, grp in envs:
specs = [s for r in grp for s in r.specs]
if any(r.url for r in grp):
continue
ex = satisfiable(specs)
if ex is None:
status = "conflict"
add("high", "PR03", ", ".join(r.src for r in grp),
"%s: no version satisfies %s%s - pip will fail with ResolutionImpossible"
% (rs[0].name, ", ".join("%s (%s)" % (",".join(map(repr, r.specs)) or "any", r.src) for r in grp if r.specs),
(" when " + m) if m else ""))
elif example is None:
example = ex
spellings = sorted({r.name for r in rs})
per_marker = defaultdict(list)
for r in decl:
per_marker[r.marker].append(r)
dups = [g for g in per_marker.values() if len(g) > 1]
if dups:
g = dups[0]
add("low", "PR04", ", ".join(r.src for r in g), "%s is declared %d times%s; keep one line"
% (rs[0].name, len(g), (" (spelt " + ", ".join(spellings) + ")") if len(spellings) > 1 else ""))
elif len(spellings) > 1:
add("info", "PR04", ", ".join(r.src for r in rs), "%s is spelt %s across files (same package after normalisation)"
% (rs[0].name, ", ".join(spellings)))
for r in decl:
if r.url:
if r.url.startswith(("git+", "hg+", "svn+", "bzr+")):
if not HEX40.search(r.url):
add("high", "PR05", r.src, "%s comes from %s without a full commit hash; the branch or tag can "
"move under you - pin @<40-char commit>" % (r.name, r.url))
elif r.url.startswith(("http://", "https://")):
if r.url.startswith("http://"):
add("high", "PR10", r.src, "%s is downloaded over plain http" % r.name)
if not r.hashes and "#sha256=" not in r.url:
add("medium", "PR05", r.src, "%s is an archive URL without a hash; add #sha256=... or --hash" % r.name)
if r.editable or r.url.startswith((".", "/", "file:")):
add("medium" if mode == "app" else "low", "PR06", r.src, "%s is %s; it will not exist on a clean build "
"machine - publish it or build a wheel" % (r.name, "an editable install" if r.editable else "a local path"))
continue
if r.editable:
add("medium", "PR06", r.src, "%s is installed editable" % r.name)
ops = {s.op for s in r.specs}
if not r.specs and key not in pinned_elsewhere:
add("high" if mode == "app" else "info", "PR01", r.src, "%s is unpinned%s" % (r.name,
"; every install can pick a new major version - pin == or add it to a lock/constraints file"
if mode == "app" else " (fine for a library; set a tested lower bound)"))
elif ops and ops <= {">=", ">", "!="} and key not in pinned_elsewhere and mode == "app":
add("medium", "PR02", r.src, "%s %s has no upper bound and no pin; a future major release installs "
"automatically" % (r.name, ",".join(map(repr, r.specs))))
if mode == "lib" and any(s.op in ("==", "===") for s in r.specs):
add("medium", "PR02", r.src, "%s is pinned == in a library; this forces the exact version on every "
"project that depends on you - use a range" % r.name)
for s in r.specs:
if s.version is not None and s.version.is_prerelease and s.op in ("==", ">=", "~="):
add("medium", "PR08", r.src, "%s %r is a pre-release" % (r.name, s))
if hash_mode and not r.hashes:
add("high", "PR07", r.src, "%s has no --hash but other lines do; pip switches to hash-checking mode "
"for the whole file and will refuse to install it" % r.name)
if hash_mode and r.hashes and not any(s.op == "==" and not s.wild for s in r.specs):
add("high", "PR07", r.src, "%s has hashes but is not pinned with ==; hash-checking mode requires "
"exact pins" % r.name)
if key in DEV_TOOLS and mode == "app" and decl and not re.search(r"dev|test|lint|ci|docs",
decl[0].src.split(":")[0], re.I):
add("low", "PR14", decl[0].src, "%s is a development/test tool in a runtime requirements file; move it "
"to requirements-dev.txt so production images stay small" % decl[0].name)
if key in DEPRECATED and decl:
sev, why = DEPRECATED[key]
add(sev, "PR11", decl[0].src, "%s %s" % (decl[0].name, why))
packages.append({"name": rs[0].name, "key": key,
"specs": ",".join(sorted({repr(s) for s in all_specs})) or ("url" if any(r.url for r in rs) else "any"),
"sources": [r.src + (" (constraint)" if r.constraint else "") for r in rs],
"marker": "; ".join(sorted({r.marker for r in rs if r.marker})),
"status": status, "example": example})
if installed:
inst = {}
with open(installed, encoding="utf-8") as f:
for n, line in enumerate(f, 1):
m = re.match(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*==\s*([^\s;#]+)", line)
if m:
inst[norm(m.group(1))] = (m.group(2), "%s:%d" % (show(installed), n))
for p in packages:
if all(r.constraint for r in by[p["key"]]):
continue # constraints never install anything
specs = [s for r in by[p["key"]] for s in r.specs]
if p["key"] not in inst:
if not any(r.marker for r in by[p["key"]]):
add("medium", "PR12", next(r.src for r in by[p["key"]] if not r.constraint), "%s is required but not in %s" % (p["name"], os.path.basename(installed)))
p["installed"] = None
continue
ver, src = inst[p["key"]]
p["installed"] = ver
try:
v = Version(ver)
except ValueError:
continue
bad = [s for s in specs if not s.contains(v)]
if bad:
add("high", "PR12", src, "%s %s is installed but the requirements say %s; the environment does not "
"match what CI builds" % (p["name"], ver, ",".join(map(repr, bad))))
order = {"high": 0, "medium": 1, "low": 2, "info": 3}
findings.sort(key=lambda f: (order[f["severity"]], f["code"], f["where"]))
counts = {k: sum(f["severity"] == k for f in findings) for k in order}
pinned = sum(1 for p in packages if re.search(r"(^|,)===?[^=,*]+(,|$)", p["specs"]) and "*" not in p["specs"])
return {"files": [show(p) for p in paths], "mode": mode, "python": python, "packages": packages,
"pinned": pinned, "findings": findings, "counts": counts, "hash_mode": hash_mode}
# ----------------------------------------------------------------------------- report
def report(r):
L = ["# Requirements check: %s" % ", ".join(r["files"]), ""]
L.append("Mode: %s%s. Packages: %d (%d pinned with ==). Findings: %d high, %d medium, %d low, %d info." % (
r["mode"], (", Python " + r["python"]) if r["python"] else "", len(r["packages"]), r["pinned"],
r["counts"]["high"], r["counts"]["medium"], r["counts"]["low"], r["counts"]["info"]))
L += ["", "**Verdict: %s**" % ("install will fail or is unsafe - fix the high findings first" if any(
f["code"] in ("PR03", "PR07", "PR13") and f["severity"] == "high" for f in r["findings"]) else
"not reproducible or risky - fix the high findings" if r["counts"]["high"] else
"reproducible and resolvable" if not r["counts"]["medium"] else "resolvable, with medium findings to tidy"), ""]
L += ["## Packages", ""]
has_inst = any("installed" in p for p in r["packages"])
L.append("| Package | Specifiers (all files) | Marker | Declared in | Status%s |" % (" | Installed" if has_inst else ""))
L.append("|---|---|---|---|---|" + ("---|" if has_inst else ""))
for p in r["packages"]:
L.append("| %s | `%s` | %s | %s | %s%s |" % (p["name"], p["specs"], ("`%s`" % p["marker"]) if p["marker"] else "-",
", ".join(p["sources"]), p["status"],
(" | %s" % (p.get("installed") or "-")) if has_inst else ""))
L += ["", "## Findings", ""]
if not r["findings"]:
L.append("- none")
for f in r["findings"]:
L.append("- **%s %s** `%s` %s" % (f["severity"].upper(), f["code"], f["where"], f["message"]))
L += ["", "## Next steps", "",
"1. Fix every PR03/PR07/PR13 line - these stop `pip install` outright.",
"2. Keep loose, intent-level ranges in `requirements.in` (or pyproject) and generate a fully pinned, hashed lock "
"with `pip-compile --generate-hashes` or `uv pip compile --generate-hashes`; install with `pip install --require-hashes -r requirements.txt`.",
"3. Use one `--index-url` (a proxy that mirrors PyPI and hosts internal packages), https only.",
"4. Run `pip-audit -r requirements.txt` (online) for known vulnerabilities - this checker is offline and does not know CVEs."]
return "\n".join(L) + "\n"
# ----------------------------------------------------------------------------- self-test
def self_test():
import tempfile
cases = []
def ok(name, cond, detail=""):
cases.append((name, bool(cond), detail))
V = Version
ok("version ordering dev < a < rc < final < post", V("1.0.dev1") < V("1.0a1") < V("1.0rc1") < V("1.0") < V("1.0.post1"))
ok("trailing zeros equal", V("1.0") == V("1.0.0"))
ok("~=1.4.2 means >=1.4.2,<1.5", Spec("~=1.4.2").contains(V("1.4.9")) and not Spec("~=1.4.2").contains(V("1.5")))
ok("~=2.1 allows 2.9 not 3.0", Spec("~=2.1").contains(V("2.9")) and not Spec("~=2.1").contains(V("3.0")))
ok("==1.4.* wildcard", Spec("==1.4.*").contains(V("1.4.7")) and not Spec("==1.4.*").contains(V("1.5")))
ok("<2.0 excludes 2.0rc1", not Spec("<2.0").contains(V("2.0rc1")))
ok("satisfiable range", satisfiable([Spec(">=1.2"), Spec("<2")]) is not None)
ok("conflict detected", satisfiable([Spec(">=2.0"), Spec("<1.9")]) is None)
ok("pin outside range conflicts", satisfiable([Spec("==1.26.4"), Spec(">=2")]) is None)
ok("!= exclusion of only pin", satisfiable([Spec("==3.1"), Spec("!=3.1")]) is None)
ok("~= vs upper bound conflict", satisfiable([Spec("~=2.31"), Spec("<2.30")]) is None)
env = marker_env("3.8", "linux")
ok("marker eval python_version", eval_marker('python_version < "3.9"', env) and
not eval_marker('python_version >= "3.9" and sys_platform == "linux"', env))
ok("marker eval or/parentheses", eval_marker('(sys_platform == "win32" or os_name == "posix")', env))
try:
Spec("~=2")
ok("~=2 rejected", False)
except ValueError:
ok("~=2 rejected", True)
d = tempfile.mkdtemp()
with open(os.path.join(d, "c.txt"), "w") as f:
f.write("urllib3<2\nnumpy==1.26.4\n")
with open(os.path.join(d, "r.txt"), "w") as f:
f.write("-c c.txt\n--extra-index-url https://pypi.internal.example/simple\nrequests==2.32.3\nurllib3>=2.2\n"
"numpy<2 ; python_version < \"3.9\"\nnumpy>=2.1 ; python_version >= \"3.9\"\nflask\n"
"Django>=4.2\ndjango<5\n-e git+https://github.com/acme/tools.git@main#egg=acme-tools\n"
"sklearn\npandas=2.2.0\npytest==8.3.3\n")
r = check([os.path.join(d, "r.txt")])
codes = defaultdict(list)
for fnd in r["findings"]:
codes[fnd["code"]].append(fnd)
ok("PR03 urllib3 conflict with constraint", any("urllib3" in x["message"] for x in codes["PR03"]), codes["PR03"])
ok("PR03 numpy conflict only for python>=3.9 marker", any("numpy" in x["message"] and "3.9" in x["message"]
for x in codes["PR03"]), codes["PR03"])
ok("PR01 flask unpinned high", any("flask" in x["message"] and x["severity"] == "high" for x in codes["PR01"]))
ok("PR04 Django/django duplicate", any("Django" in x["message"] for x in codes["PR04"]))
ok("PR05 git branch ref", any("acme-tools" in x["message"] for x in codes["PR05"]), codes["PR05"])
ok("PR09 extra index", bool(codes["PR09"]))
ok("PR11 sklearn", any("scikit-learn" in x["message"] for x in codes["PR11"]))
ok("PR14 pytest in runtime file", any("pytest" in x["message"] for x in codes["PR14"]))
ok("PR13 single =", any("pandas" in x["message"] for x in codes["PR13"]))
r38 = check([os.path.join(d, "r.txt")], python="3.8")
ok("--python 3.8 drops the >=3.9 numpy line", not any("numpy" in x["message"] for x in r38["findings"] if x["code"] == "PR03"))
with open(os.path.join(d, "h.txt"), "w") as f:
f.write("six==1.16.0 --hash=sha256:" + "a" * 64 + "\nidna>=3 --hash=sha256:" + "b" * 64 + "\ncertifi==2024.8.30\n")
rh = check([os.path.join(d, "h.txt")])
ok("PR07 partial hashes", sum(x["code"] == "PR07" for x in rh["findings"]) == 2, rh["findings"])
with open(os.path.join(d, "freeze.txt"), "w") as f:
f.write("six==1.15.0\n")
rf = check([os.path.join(d, "h.txt")], installed=os.path.join(d, "freeze.txt"))
ok("PR12 installed mismatch", any(x["code"] == "PR12" and "six" in x["message"] for x in rf["findings"]))
rl = check([os.path.join(d, "h.txt")], mode="lib")
ok("lib mode flags == pin", any(x["code"] == "PR02" and "library" in x["message"] for x in rl["findings"]))
ok("report renders", "Verdict" in report(r) and "| Package |" in report(r))
for name, good, detail in cases:
print("%s %s%s" % ("PASS" if good else "FAIL", name, "" if good else " -> %r" % (detail,)))
passed = sum(1 for _, g, _ in cases if g)
print("%d/%d passed" % (passed, len(cases)))
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check pip requirements files for pins, conflicts and risky sources.")
ap.add_argument("files", nargs="*")
ap.add_argument("--mode", choices=("app", "lib"), default="app",
help="app = deployable service (pins expected), lib = package others install (ranges expected)")
ap.add_argument("--installed", help="pip freeze output to compare against")
ap.add_argument("--python", help="evaluate markers for this Python version, e.g. 3.11")
ap.add_argument("--platform", help="linux | windows | macos (with --python)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.files:
ap.print_usage(sys.stderr)
return 2
try:
r = check(a.files, a.mode, a.installed, a.python, a.platform)
except (OSError, UnicodeDecodeError) as e:
print("error: %s" % e, file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
return 1 if r["counts"]["high"] else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
---
name: api-rate-limit-policy-simulator
description: Simulates API rate limits before you ship them - token bucket, fixed window, sliding log and sliding counter, per client, endpoint, tier or global - against generated or recorded traffic with retrying clients. Shows who gets 429s and for how long, fixed-window boundary bursts, retry herds, noisy neighbours, abusive clients that still get through and peaks over backend capacity. Includes an offline stdlib Python simulator with --self-test.
---
# API Rate Limit Policy Simulator
You help backend and platform engineers choose rate limits with evidence instead of round numbers. A limit that looks safe on paper ("300 per minute per key") can still let 1.8x that through at a window boundary, throttle a mobile app whose thousands of users share one API key, release a herd of retries at the reset second, and do nothing against a scraper that stays just under it. You describe the policies and the traffic in a small JSON file, replay it, and read the result client by client and policy by policy.
## Files in this skill
- `scripts/rate_sim.py` - simulates token bucket, fixed window, sliding log and sliding counter limiters keyed by client, endpoint, client+endpoint or global, scoped to endpoints and client tiers; generates steady, Poisson or burst traffic (or replays a CSV trace), lets clients retry immediately, with backoff or by honouring Retry-After, and reports 429s, delays, failures, worst-case and observed window counts, peaks vs capacity, findings RL01-RL10 and a sample 429 response (Python 3.8+ standard library only)
- `references/algorithms.md` - how each algorithm behaves, its worst case, memory cost and when to pick it
- `references/policy-design.md` - choosing keys, tiers, numbers, headers and client retry rules; what the finding codes mean
- `templates/rate-limit-review.md` - the review to hand back
- `examples/orders-api.json` - a public orders API with a per-key fixed window, a login limit and a global fixed window; five clients including a shared mobile key, a bulk partner, a scraper and a credential stuffer
- `examples/orders-api-report.md` - the real simulator output on that file
- `examples/orders-api-tuned.json` - the same API after the fixes: per-tier token buckets, a global token bucket sized to capacity
## Workflow
### 1. Collect the inputs
Ask for: the limits as configured today or as proposed (algorithm, key, numbers, which endpoints and which clients), where they are enforced (gateway, CDN, app, Redis), the backend capacity in requests per second it can sustain, and the traffic: the biggest legitimate clients and their shape (steady, bursty batch jobs, many users behind one key or one NAT IP), how each one retries, and any abuse seen in logs. Real access logs beat guesses: export `t,client,endpoint` for a busy 5-15 minutes and pass it with `--trace`.
### 2. Run the simulator
```
python3 scripts/rate_sim.py examples/orders-api.json
python3 scripts/rate_sim.py examples/orders-api-tuned.json
python3 scripts/rate_sim.py policies.json --trace access-sample.csv
python3 scripts/rate_sim.py policies.json --json
python3 scripts/rate_sim.py --self-test
```
Exit 0 = no high findings, 1 = high findings (legitimate clients throttled or admitted traffic over capacity), 2 = bad input. Without a shell, reason with `references/algorithms.md`: per key, compare the client's peak per window with the limit, the worst case (2x limit for fixed windows, burst + rate for buckets) with capacity, and think through what each client does after a 429.
### 3. Read the result
- Legit clients first (RL01): any failed or delayed requests, and how long they waited. One key shared by an app's users needs its own tier or a per-user key.
- Capacity (RL03): the peak admitted second vs what the backend sustains. A global limit is the only thing that protects capacity; per-key limits multiply by the number of keys.
- Shape (RL04, RL10): fixed windows let bursts through at boundaries and hand out a whole window at once; retries without jitter all come back in the same second.
- Abuse (RL02): a scraper under the per-key limit is not stopped by limits; it needs detection, auth or a per-endpoint limit.
- Retries (RL05): attempts per logical request above 1.5 means the clients amplify load when throttled.
### 4. Tune and re-run
Change one thing at a time and re-run with the same traffic: switch fixed windows to token buckets or sliding counters, add tiers with `clients` / `exclude_clients`, size the global bucket to capacity (rate = sustainable rps, burst = what the backend absorbs for a second), add endpoint limits for login and expensive calls, and agree pacing with batch partners. Use `references/policy-design.md` for the numbers.
### 5. Deliver
Fill `templates/rate-limit-review.md`: verdict, a before/after table per client, the policy changes as config, the 429 response and headers, client guidance (Retry-After, backoff with jitter, max retries) and what to monitor after rollout.
## Example result
On `examples/orders-api.json` the shared `mobile-app` key hits the 300/minute per-key window: 1,042 of its 1,839 requests succeed only after waiting up to 60 s (RL01, high). Honouring Retry-After without jitter releases 289 retries in second 240, so admitted traffic peaks at 313 req/s against a capacity of 80 (RL03, RL10). The per-key fixed window lets 544 requests through in one rolling minute (1.8x, RL04), and the scraper still gets 45% of its calls through while its immediate retries push it to 2.67 attempts per request. The login limit stops the credential stuffer (40 of 480). In `examples/orders-api-tuned.json` (tier token buckets, global bucket 50 req/s + 30 burst, partner pacing at 2 req/s) legit clients see no 429s, the peak is 27 req/s and only the scraper findings remain.
## Rules
- Simulate with the traffic you actually have (or a trace), not an average; bursts and retries are where limits fail.
- Never call a limit safe for capacity unless a global limit bounds the total.
- Treat delayed requests of legitimate clients as harm, not as success.
- Always return 429 with Retry-After, add jitter, and tell clients how to back off.
- Rate limits are not abuse protection on their own; say when authentication, bot detection or WAF rules are needed.
- Results are a model: confirm with gateway metrics after rollout and keep the limits in config, not code.
FILE:references/algorithms.md
# Rate limit algorithms
| Algorithm | Admits | Worst case in one window | Memory per key | Good for |
|---|---|---|---|---|
| Token bucket (GCRA) | while tokens remain; refills at `rate`/s up to `burst` | burst + rate x window | 2 numbers | most APIs: smooth average, bounded bursts |
| Fixed window | `limit` per calendar window, counter resets | 2 x limit (end of one window + start of next) | 1 counter | quotas and billing ("10,000 per day"), not capacity |
| Sliding log | at most `limit` in any rolling window | limit (exact) | one timestamp per request | small limits: login, OTP, password reset |
| Sliding counter | previous window weighted by overlap + current | about limit (assumes even spread) | 2 counters | large limits where exactness is not needed |
## Token bucket
- `rate` is the long-run average; `burst` is how many calls can arrive at once after a quiet period.
- `limit` + `window_s` without a burst means rate = limit/window and burst = limit: a client can spend the whole minute in one second. Set the burst explicitly.
- Retry-After = (cost - tokens) / rate: usually well under a second, so retries spread out by themselves.
- GCRA (used by many gateways) stores one timestamp and gives the same admit/reject decisions.
## Fixed window
- Cheap (`INCR` + `EXPIRE` in Redis) and easy to explain, so it is everywhere.
- A client can send `limit` in the last second of a window and `limit` again in the first second of the next.
- Every throttled client gets Retry-After = time to the reset, so they all come back in the same second (retry herd).
- Use it for quotas that match business periods; put a token bucket in front of it for capacity.
## Sliding log
- Exact, but stores every admitted timestamp for the window: 10,000/hour per key = 10,000 entries per key.
- Retry-After = oldest relevant entry + window - now: can be long; tell the client.
## Sliding counter
- Estimate = previous count x (1 - elapsed/window) + current count.
- Error comes from uneven traffic: a burst at the end of the previous window is under-counted later (in the simulator's self-test, 10 requests at the very end of one window plus an even trickle later admit 16 in one rolling minute for a limit of 10).
- Good default for large per-key limits behind a token bucket.
## Concurrency limits (not simulated)
Limits on in-flight requests protect slow endpoints better than rate limits; combine with a rate limit for fairness.
FILE:references/policy-design.md
# Designing rate limit policies
## Pick the key
- **API key / token** for authenticated traffic. Watch for keys shared by many users (mobile apps, SDKs embedded in customer sites): give them a tier or move to per-user keys.
- **User or account id** behind the key when the API knows it; fairer than the key.
- **IP** only for anonymous endpoints; offices, schools, carriers and VPNs put thousands of users behind one IP (CGNAT). Use generous IP limits and stricter account limits.
- **Endpoint** for expensive calls (search, export, bulk) and sensitive ones (login, OTP, password reset).
- **Global** to protect capacity. Without it, per-key limits x number of keys is the real ceiling.
## Pick the numbers
1. Sustained rate per key: p99 of a normal busy client's requests per second x 2-3 headroom.
2. Burst: the largest legitimate batch (page load fan-out, sync job) that must not be throttled.
3. Global: rate = what the backend sustains at acceptable latency (load test), burst = what it absorbs for about a second.
4. Login/OTP: 5-10 per minute per account and per IP, sliding log, plus lockout or CAPTCHA after repeated failures.
5. Quotas (per day/month) are a product decision; keep them separate from capacity limits.
## Tiers
Scope policies with `clients` and `exclude_clients` in the simulator; in production map tiers to plans or key metadata (free, paid, partner, internal). Internal services should usually bypass public limits but stay under the global one.
## Responses and headers
- 429 Too Many Requests, `Retry-After` in seconds, a problem+json body naming the limit.
- Advertise the policy: the IETF httpapi ratelimit-headers draft defines `RateLimit-Policy` and `RateLimit` fields (field syntax changed between drafts); many gateways still send `X-RateLimit-Limit`, `-Remaining`, `-Reset`. Pick one and document it.
- Add 0-20% random jitter to Retry-After for fixed windows so throttled clients do not return together.
- Use 503 + Retry-After for overload (global limit), 429 for per-client limits, so clients can tell them apart.
## Client rules to publish
Honour Retry-After; otherwise exponential backoff with full jitter (1, 2, 4 s ...), at most 3-5 retries, never retry non-idempotent calls without an idempotency key; pace batch jobs instead of bursting.
## Finding codes (rate_sim.py)
| Code | Meaning |
|---|---|
| RL01 | legit client failed or delayed (high above 1% of its requests) |
| RL02 | abusive client still gets more than 25% through |
| RL03 | admitted peak over capacity, or no global limit to bound it |
| RL04 | fixed window let more than the limit through in a rolling window |
| RL05 | client retries amplify attempts 1.5x or more |
| RL06 | noisy neighbour holds most of a global limit while legit clients are rejected |
| RL07 | sliding log with a very large limit (memory) |
| RL08 | reminder: 429 + Retry-After and documented limits |
| RL09 | policy never triggers and is 10x above the busiest key |
| RL10 | retry herd: most of the peak second are retries released together |
FILE:templates/rate-limit-review.md
# Rate limit review: <service>
**Scope:** <endpoints / gateway> **Capacity:** <80 req/s sustained> **Traffic used:** <generated from ... / trace of 10 min on ...>
**Verdict:** <ship / ship with changes / do not ship>
## What happens today
| Client | Kind | Requests | 429s | OK after retry | Failed | Max wait | Main policy |
|---|---|---|---|---|---|---|---|
| <mobile-app> | <legit> | <1,839> | <1,081> | <1,042> | <0> | <60 s> | <per-key minute> |
Peak admitted: <313 req/s in second 240> vs capacity <80>.
## Findings
1. <HIGH RL01 - shared mobile key throttled; 57% of requests delayed up to 60 s>
2. <HIGH RL03 / MEDIUM RL10 - retry herd at window reset, 4x capacity>
3. <MEDIUM RL04 - fixed window 1.8x at boundary>
4. <MEDIUM RL02 - scraper under the per-key limit>
## Proposed policies
```json
<policies block from the tuned config>
```
## After the change (same traffic)
| Client | 429s before | 429s after | Failed before | Failed after |
|---|---|---|---|---|
| <...> | | | | |
Peak admitted after: <27 req/s>.
## Response contract
- 429 + Retry-After (+ jitter) for per-client limits; 503 + Retry-After for global overload
- Headers: <RateLimit-Policy / RateLimit or X-RateLimit-*>
- Client guidance: <honour Retry-After, backoff with jitter, max 3 retries, pace batch jobs>
## Outside rate limits
<bot detection / auth / WAF rule for the scraper; lockout or CAPTCHA for login>
## Monitor after rollout
<429 rate per client and policy, top throttled keys, p99 latency, admitted rps vs capacity; review in 2 weeks>
FILE:examples/orders-api.json
{
"service": "Orders API (public v2)",
"duration_s": 300,
"capacity_rps": 80,
"seed": 7,
"policies": [
{"name": "per-key minute", "algorithm": "fixed_window", "key": "client", "limit": 300, "window_s": 60},
{"name": "login", "algorithm": "sliding_log", "key": "client", "limit": 10, "window_s": 60,
"endpoints": ["/v2/login"]},
{"name": "global", "algorithm": "fixed_window", "key": "global", "limit": 3600, "window_s": 60}
],
"clients": [
{"name": "mobile-app", "kind": "legit", "pattern": "poisson", "rps": 6, "endpoint": "/v2/orders",
"retry": "honor"},
{"name": "partner-erp", "kind": "legit", "pattern": "burst", "burst": 120, "every_s": 60, "start": 58,
"endpoint": "/v2/orders/bulk", "retry": "backoff"},
{"name": "checkout-web", "kind": "legit", "pattern": "poisson", "rps": 3, "endpoint": "/v2/checkout",
"retry": "honor"},
{"name": "price-scraper", "kind": "abusive", "pattern": "steady", "rps": 12, "start": 20,
"endpoint": "/v2/products", "retry": "immediate"},
{"name": "cred-stuffer", "kind": "abusive", "pattern": "steady", "rps": 2, "start": 60,
"endpoint": "/v2/login"}
]
}
FILE:examples/orders-api-report.md
# Rate limit simulation: Orders API (public v2)
Simulated 300s, 7227 logical requests, 4930 admitted, peak 313 req/s admitted (second 240) vs capacity 80.
**Verdict: 2 high finding(s) - fix before shipping these limits**
## Clients
| Client | Kind | Requests | Attempts | Allowed | 429s | OK after retry | Failed | Failed % | Max Retry-After | Top policy |
|---|---|---|---|---|---|---|---|---|---|---|
| price-scraper | abusive | 3360 | 8961 | 1503 | 7458 | 15 | 1857 | 55.3% | 35.2s | per-key minute |
| mobile-app | legit | 1839 | 2920 | 1839 | 1081 | 1042 | 0 | 0.0% | 59.9s | per-key minute |
| checkout-web | legit | 948 | 948 | 948 | 0 | 0 | 0 | 0.0% | 0s | - |
| partner-erp | legit | 600 | 600 | 600 | 0 | 0 | 0 | 0.0% | 0s | - |
| cred-stuffer | abusive | 480 | 480 | 40 | 440 | 0 | 440 | 91.7% | 55s | login |
## Policies
| Policy | Algorithm | Key | Scope | Sustained | Worst case / window | Busiest key / window | Checks | Rejects |
|---|---|---|---|---|---|---|---|---|
| per-key minute | fixed_window | client | all | 5 req/s | 600 per 60s | 544 | 13909 | 8539 |
| login | sliding_log | client | /v2/login | 0.167 req/s | 10 per 60s | 10 | 480 | 440 |
| global | fixed_window | global | all | 60 req/s | 7200 per 60s | 1235 | 13909 | 0 |
## Findings
- **HIGH RL01** Legit client mobile-app is throttled on 56.7% of its requests (0 failed, 1042 only after retrying, waits up to 59.9s), mostly by policy 'per-key minute'; raise that limit or give this client its own tier.
- **HIGH RL03** Admitted traffic peaks at 313 req/s (second 240) over the backend capacity of 80 req/s; add or lower a global limit.
- **MEDIUM RL02** Abusive client price-scraper still gets 45% of its requests through (1503); tighten the per-client limit or add a stricter policy for its pattern.
- **MEDIUM RL04** Fixed window 'per-key minute' let 544 requests through in one rolling 60s window (limit 300, 1.8x) across a window boundary; use sliding_counter or a token bucket if the limit protects capacity.
- **MEDIUM RL05** Client price-scraper sends 2.67x attempts per logical request because it retries throttled calls; retries count against the limit and add load - require Retry-After or backoff with jitter.
- **MEDIUM RL05** Client mobile-app sends 1.59x attempts per logical request because it retries throttled calls; retries count against the limit and add load - require Retry-After or backoff with jitter.
- **MEDIUM RL10** Retry herd: 289 of the 313 requests admitted in second 240 are retries released by the same Retry-After / window reset; add jitter to Retry-After (or client backoff) and prefer a token bucket or sliding window so capacity is not handed out all at once.
- **INFO RL08** Return 429 with Retry-After on every rejection (see the sample headers) and document the limits so clients can pace themselves.
## Sample 429 (first rejection, t=45.0s, price-scraper /v2/products)
```
HTTP/1.1 429 Too Many Requests
Retry-After: 15
RateLimit-Policy: "per-key minute";q=300;w=60
RateLimit: "per-key minute";r=0;t=15
Content-Type: application/problem+json
```
Header names follow the IETF httpapi ratelimit-headers draft; older gateways use X-RateLimit-Limit / -Remaining / -Reset instead.
FILE:examples/orders-api-tuned.json
{
"service": "Orders API (public v2, tuned)",
"notes": "Same traffic as orders-api.json except partner-erp, which agreed to pace its bulk sync at 2 req/s instead of 120-request bursts. Token buckets per tier, a global token bucket sized to capacity.",
"duration_s": 300,
"capacity_rps": 80,
"seed": 7,
"policies": [
{"name": "default key", "algorithm": "token_bucket", "key": "client", "rate": 4, "burst": 40,
"exclude_clients": ["mobile-app", "partner-erp"]},
{"name": "mobile tier", "algorithm": "token_bucket", "key": "client", "rate": 15, "burst": 60,
"clients": ["mobile-app"]},
{"name": "partner tier", "algorithm": "token_bucket", "key": "client", "rate": 3, "burst": 20,
"clients": ["partner-erp"]},
{"name": "login", "algorithm": "sliding_log", "key": "client", "limit": 10, "window_s": 60,
"endpoints": ["/v2/login"]},
{"name": "global", "algorithm": "token_bucket", "key": "global", "rate": 50, "burst": 30}
],
"clients": [
{"name": "mobile-app", "kind": "legit", "pattern": "poisson", "rps": 6, "endpoint": "/v2/orders",
"retry": "honor"},
{"name": "partner-erp", "kind": "legit", "pattern": "steady", "rps": 2, "start": 58,
"endpoint": "/v2/orders/bulk", "retry": "backoff"},
{"name": "checkout-web", "kind": "legit", "pattern": "poisson", "rps": 3, "endpoint": "/v2/checkout",
"retry": "honor"},
{"name": "price-scraper", "kind": "abusive", "pattern": "steady", "rps": 12, "start": 20,
"endpoint": "/v2/products", "retry": "immediate"},
{"name": "cred-stuffer", "kind": "abusive", "pattern": "steady", "rps": 2, "start": 60,
"endpoint": "/v2/login"}
]
}
FILE:scripts/rate_sim.py
#!/usr/bin/env python3
"""API rate limit policy simulator (Python 3.8+ standard library only).
Replays generated or recorded traffic against one or more rate limit policies and shows who
gets throttled, why, for how long, and whether the policies protect the backend without
hurting legitimate clients.
Algorithms (per policy):
token_bucket rate (tokens/s) + burst (bucket size); or limit + window_s (rate = limit/window_s,
burst defaults to limit). GCRA behaves the same for admit/reject.
fixed_window limit requests per window_s, counter resets on window boundaries
(allows up to 2 x limit across a boundary)
sliding_log exact: at most limit requests in any window_s (stores one timestamp per request)
sliding_counter approximation: previous window weighted by overlap + current window (Cloudflare /
Redis style, about limit per rolling window, a few % error)
Config JSON:
{
"service": "Orders API", "duration_s": 300, "capacity_rps": 60, "seed": 7,
"policies": [
{"name": "per-client burst", "algorithm": "token_bucket", "key": "client", "rate": 5, "burst": 20},
{"name": "per-client minute", "algorithm": "fixed_window", "key": "client", "limit": 200, "window_s": 60},
{"name": "login", "algorithm": "sliding_log", "key": "client", "limit": 5, "window_s": 60,
"endpoints": ["/login"]},
{"name": "global", "algorithm": "sliding_counter", "key": "global", "limit": 3000, "window_s": 60}
],
"clients": [
{"name": "mobile-app", "kind": "legit", "pattern": "steady", "rps": 2, "endpoint": "/orders"},
{"name": "nightly-sync", "kind": "legit", "pattern": "burst", "burst": 40, "every_s": 60, "start": 5},
{"name": "scraper", "kind": "abusive", "pattern": "poisson", "rps": 25, "retry": "immediate"}
]
}
key client | endpoint | client+endpoint | global
endpoints optional list: the policy applies only to these endpoints (prefix match)
clients / exclude_clients optional lists: apply only to / skip these clients (tiers, allow lists)
pattern steady (rps) | poisson (rps, seeded) | burst (burst requests every every_s seconds)
start, duration seconds (default 0 and duration_s); cost = tokens per request (default 1)
retry none (default) | immediate (0.1 s later) | backoff (1, 2, 4 s) | honor (Retry-After);
max_retries default 3
kind legit | abusive | unknown - used to judge false positives and leakage
Recorded traffic instead of generated: --trace trace.csv with columns t,client[,endpoint][,cost]
(t in seconds from start). Clients in the config then only add kind/retry information.
Usage:
python3 rate_sim.py config.json [--trace trace.csv] [--json]
python3 rate_sim.py --self-test
Exit codes: 0 = no high findings, 1 = high findings (legit clients throttled or backend over
capacity), 2 = bad input.
"""
import argparse
import csv
import heapq
import json
import math
import random
import sys
from collections import defaultdict, deque
ALGOS = ("token_bucket", "fixed_window", "sliding_log", "sliding_counter")
KEYS = ("client", "endpoint", "client+endpoint", "global")
class InputError(ValueError):
pass
# ----------------------------------------------------------------------------- limiters
class TokenBucket:
def __init__(self, rate, burst):
self.rate, self.burst = float(rate), float(burst)
self.state = {}
def check(self, key, t, cost):
tokens, last = self.state.get(key, (self.burst, t))
tokens = min(self.burst, tokens + (t - last) * self.rate)
self.state[key] = (tokens, t)
if tokens + 1e-9 >= cost:
return True, 0.0, tokens
return False, (cost - tokens) / self.rate, tokens
def consume(self, key, t, cost):
tokens, last = self.state[key]
self.state[key] = (tokens - cost, t)
class FixedWindow:
def __init__(self, limit, window):
self.limit, self.window = float(limit), float(window)
self.state = {}
def check(self, key, t, cost):
w = math.floor(t / self.window)
start, count = self.state.get(key, (w, 0.0))
if start != w:
start, count = w, 0.0
self.state[key] = (start, count)
if count + cost <= self.limit + 1e-9:
return True, 0.0, self.limit - count
return False, (w + 1) * self.window - t, self.limit - count
def consume(self, key, t, cost):
start, count = self.state[key]
self.state[key] = (start, count + cost)
class SlidingLog:
def __init__(self, limit, window):
self.limit, self.window = float(limit), float(window)
self.state = defaultdict(deque)
def check(self, key, t, cost):
log = self.state[key]
while log and log[0] <= t - self.window:
log.popleft()
used = len(log)
if used + cost <= self.limit + 1e-9:
return True, 0.0, self.limit - used
need = int(used + cost - self.limit) # entries that must expire first
idx = min(max(need - 1, 0), used - 1) if used else 0
wait = (log[idx] + self.window - t) if used else self.window
return False, max(wait, 0.001), self.limit - used
def consume(self, key, t, cost):
for _ in range(int(round(cost))):
self.state[key].append(t)
class SlidingCounter:
def __init__(self, limit, window):
self.limit, self.window = float(limit), float(window)
self.state = {}
def _roll(self, key, t):
w = math.floor(t / self.window)
start, prev, curr = self.state.get(key, (w, 0.0, 0.0))
if w == start + 1:
start, prev, curr = w, curr, 0.0
elif w > start + 1:
start, prev, curr = w, 0.0, 0.0
self.state[key] = (start, prev, curr)
return start, prev, curr
def check(self, key, t, cost):
start, prev, curr = self._roll(key, t)
e = t - start * self.window
est = prev * (1 - e / self.window) + curr
if est + cost <= self.limit + 1e-9:
return True, 0.0, self.limit - est
lim, w = self.limit, self.window
if curr + cost <= lim and prev > 0:
wait = w * (1 - (lim - curr - cost) / prev) - e
else:
x = 0.0
if curr > 0 and lim - cost < curr:
x = w * (1 - (lim - cost) / curr)
wait = (w - e) + max(0.0, x)
return False, max(wait, 0.001), self.limit - est
def consume(self, key, t, cost):
start, prev, curr = self.state[key]
self.state[key] = (start, prev, curr + cost)
def make_limiter(p):
algo = p["algorithm"]
if algo == "token_bucket":
return TokenBucket(p["rate"], p["burst"])
if algo == "fixed_window":
return FixedWindow(p["limit"], p["window_s"])
if algo == "sliding_log":
return SlidingLog(p["limit"], p["window_s"])
return SlidingCounter(p["limit"], p["window_s"])
# ----------------------------------------------------------------------------- config
def num(v, what, positive=True):
try:
x = float(v)
except (TypeError, ValueError):
raise InputError("%s must be a number, got %r" % (what, v))
if positive and x <= 0:
raise InputError("%s must be > 0, got %r" % (what, v))
return x
def normalize(cfg):
if not isinstance(cfg, dict):
raise InputError("config must be a JSON object")
pols = cfg.get("policies")
if not pols:
raise InputError("config needs a non-empty 'policies' list")
out = []
for i, p in enumerate(pols, 1):
q = dict(p)
q.setdefault("name", "policy-%d" % i)
algo = q.get("algorithm")
if algo not in ALGOS:
raise InputError("policy %r: algorithm must be one of %s" % (q["name"], ", ".join(ALGOS)))
q.setdefault("key", "client")
if q["key"] not in KEYS:
raise InputError("policy %r: key must be one of %s" % (q["name"], ", ".join(KEYS)))
if algo == "token_bucket":
if "rate" in q:
q["rate"] = num(q["rate"], q["name"] + ".rate")
q["burst"] = num(q.get("burst", max(1.0, q["rate"])), q["name"] + ".burst")
elif "limit" in q:
lim = num(q["limit"], q["name"] + ".limit")
win = num(q.get("window_s", 1), q["name"] + ".window_s")
q["rate"] = lim / win
q["burst"] = num(q.get("burst", lim), q["name"] + ".burst")
else:
raise InputError("policy %r: token_bucket needs rate (+burst) or limit + window_s" % q["name"])
q.setdefault("window_s", 1.0)
q["window_s"] = num(q["window_s"], q["name"] + ".window_s")
else:
q["limit"] = num(q.get("limit"), q["name"] + ".limit")
q["window_s"] = num(q.get("window_s"), q["name"] + ".window_s")
q["endpoints"] = list(q.get("endpoints") or [])
q["clients"] = list(q.get("clients") or [])
q["exclude_clients"] = list(q.get("exclude_clients") or [])
out.append(q)
cfg = dict(cfg)
cfg["policies"] = out
cfg["duration_s"] = num(cfg.get("duration_s", 300), "duration_s")
if cfg.get("capacity_rps") is not None:
cfg["capacity_rps"] = num(cfg["capacity_rps"], "capacity_rps")
clients = []
for i, c in enumerate(cfg.get("clients") or [], 1):
c = dict(c)
c.setdefault("name", "client-%d" % i)
c.setdefault("kind", "unknown")
c.setdefault("pattern", "steady")
c.setdefault("endpoint", "/")
c.setdefault("retry", "none")
if c["retry"] not in ("none", "immediate", "backoff", "honor"):
raise InputError("client %r: retry must be none, immediate, backoff or honor" % c["name"])
c["max_retries"] = int(c.get("max_retries", 3))
c["cost"] = num(c.get("cost", 1), c["name"] + ".cost")
c["start"] = num(c.get("start", 0), c["name"] + ".start", positive=False)
c["duration"] = num(c.get("duration", cfg["duration_s"] - c["start"]), c["name"] + ".duration")
if c["pattern"] in ("steady", "poisson"):
c["rps"] = num(c.get("rps"), c["name"] + ".rps")
elif c["pattern"] == "burst":
c["burst"] = int(num(c.get("burst"), c["name"] + ".burst"))
c["every_s"] = num(c.get("every_s", 60), c["name"] + ".every_s")
else:
raise InputError("client %r: pattern must be steady, poisson or burst" % c["name"])
clients.append(c)
cfg["clients"] = clients
return cfg
def generate(cfg):
rng = random.Random(cfg.get("seed", 1))
reqs = []
for c in cfg["clients"]:
t0, t1 = c["start"], c["start"] + c["duration"]
if c["pattern"] == "steady":
n = int(round(c["duration"] * c["rps"]))
reqs += [(t0 + i / c["rps"], c["name"], c["endpoint"], c["cost"]) for i in range(n)]
elif c["pattern"] == "poisson":
t = t0 + rng.expovariate(c["rps"])
while t < t1:
reqs.append((t, c["name"], c["endpoint"], c["cost"]))
t += rng.expovariate(c["rps"])
else:
t = t0
while t < t1:
reqs += [(t + k * 0.001, c["name"], c["endpoint"], c["cost"]) for k in range(c["burst"])]
t += c["every_s"]
return reqs
def read_trace(path):
reqs = []
with open(path, newline="", encoding="utf-8") as f:
for n, row in enumerate(csv.DictReader(f), 2):
try:
reqs.append((float(row["t"]), row["client"].strip(), (row.get("endpoint") or "/").strip(),
float(row.get("cost") or 1)))
except (KeyError, ValueError, AttributeError):
raise InputError("%s line %d: need t,client[,endpoint][,cost]" % (path, n))
return reqs
# ----------------------------------------------------------------------------- simulation
def policy_key(p, client, endpoint):
if p["endpoints"] and not any(endpoint.startswith(e) for e in p["endpoints"]):
return None
if (p["clients"] and client not in p["clients"]) or client in p["exclude_clients"]:
return None
return {"client": client, "endpoint": endpoint, "client+endpoint": client + " " + endpoint,
"global": "*"}[p["key"]]
def simulate(cfg, reqs):
pols = cfg["policies"]
lims = [make_limiter(p) for p in pols]
cinfo = {c["name"]: c for c in cfg["clients"]}
heap = []
for i, (t, cl, ep, cost) in enumerate(sorted(reqs)):
heapq.heappush(heap, (t, i, cl, ep, cost, 0, i))
seq = len(reqs)
stats = defaultdict(lambda: {"logical": 0, "attempts": 0, "allowed": 0, "rejected": 0, "failed": 0, "delayed": 0,
"max_retry_after": 0.0, "blocked_by": defaultdict(int), "first_reject": None})
pstats = [{"checks": 0, "rejects": 0, "allowed_times": defaultdict(list), "by_client": defaultdict(int)}
for _ in pols]
admitted = []
sample = None
while heap:
t, _, cl, ep, cost, attempt, lid = heapq.heappop(heap)
s = stats[cl]
if attempt == 0:
s["logical"] += 1
s["attempts"] += 1
keys = [policy_key(p, cl, ep) for p in pols]
rejecting, wait, remain = [], 0.0, {}
for j, (lim, k) in enumerate(zip(lims, keys)):
if k is None:
continue
pstats[j]["checks"] += 1
ok, w, rem = lim.check(k, t, cost)
remain[j] = rem
if not ok:
rejecting.append(j)
wait = max(wait, w)
if not rejecting:
for j, (lim, k) in enumerate(zip(lims, keys)):
if k is not None:
lim.consume(k, t, cost)
pstats[j]["allowed_times"][k].append(t)
s["allowed"] += 1
if attempt:
s["delayed"] += 1
admitted.append((t, cl, attempt))
continue
s["rejected"] += 1
s["max_retry_after"] = max(s["max_retry_after"], wait)
if s["first_reject"] is None:
s["first_reject"] = t
s["blocked_by"][pols[rejecting[0]]["name"]] += 1
for j in rejecting:
pstats[j]["rejects"] += 1
pstats[j]["by_client"][cl] += 1
if sample is None:
j = rejecting[0]
sample = {"t": round(t, 3), "client": cl, "endpoint": ep, "policy": pols[j]["name"],
"retry_after": max(1, math.ceil(wait)), "quota": pols[j].get("limit", pols[j].get("burst")),
"window_s": pols[j]["window_s"]}
c = cinfo.get(cl, {})
mode, maxr = c.get("retry", "none"), c.get("max_retries", 3)
if mode != "none" and attempt < maxr:
delay = {"immediate": 0.1, "backoff": 2.0 ** attempt, "honor": wait + 0.05}[mode]
seq += 1
heapq.heappush(heap, (t + delay, seq, cl, ep, cost, attempt + 1, lid))
else:
s["failed"] += 1
return stats, pstats, admitted, sample
def max_in_window(times, w):
best, dq = 0, deque()
for t in sorted(times):
dq.append(t)
while dq[0] <= t - w:
dq.popleft()
best = max(best, len(dq))
return best
def theoretical_max(p):
w = p["window_s"]
if p["algorithm"] == "token_bucket":
return p["burst"] + p["rate"] * w
if p["algorithm"] == "fixed_window":
return 2 * p["limit"]
return p["limit"]
def sustained_rps(p):
return p["rate"] if p["algorithm"] == "token_bucket" else p["limit"] / p["window_s"]
# ----------------------------------------------------------------------------- analysis
def analyse(cfg, reqs):
stats, pstats, admitted, sample = simulate(cfg, reqs)
pols = cfg["policies"]
kinds = {c["name"]: c["kind"] for c in cfg["clients"]}
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
per_sec = defaultdict(int)
retry_sec = defaultdict(int)
for t, _, att in admitted:
per_sec[int(t)] += 1
if att:
retry_sec[int(t)] += 1
peak_rps = max(per_sec.values()) if per_sec else 0
peak_sec = max(per_sec, key=per_sec.get) if per_sec else 0
clients = []
for name in sorted(stats, key=lambda n: (-stats[n]["logical"], n)):
s = stats[name]
fail_pct = 100.0 * s["failed"] / s["logical"] if s["logical"] else 0.0
hurt_pct = 100.0 * (s["failed"] + s["delayed"]) / s["logical"] if s["logical"] else 0.0
top = max(s["blocked_by"], key=s["blocked_by"].get) if s["blocked_by"] else ""
clients.append({"client": name, "kind": kinds.get(name, "unknown"), "logical": s["logical"],
"attempts": s["attempts"], "allowed": s["allowed"], "rejected": s["rejected"],
"failed": s["failed"], "failed_pct": round(fail_pct, 1),
"delayed": s["delayed"],
"amplification": round(s["attempts"] / s["logical"], 2) if s["logical"] else 0,
"max_retry_after_s": round(s["max_retry_after"], 1), "top_policy": top,
"first_reject_s": None if s["first_reject"] is None else round(s["first_reject"], 1)})
k = kinds.get(name, "unknown")
if k == "legit" and hurt_pct > 1.0:
add("high", "RL01", "Legit client %s is throttled on %.1f%% of its requests (%d failed, %d only after "
"retrying, waits up to %.1fs), mostly by policy '%s'; raise that limit or give this client its own tier."
% (name, hurt_pct, s["failed"], s["delayed"], s["max_retry_after"], top))
elif k == "legit" and s["rejected"]:
add("low", "RL01", "Legit client %s sees %d 429s but its retries recover; check it handles Retry-After."
% (name, s["rejected"]))
if k == "abusive" and s["logical"]:
share = 100.0 * s["allowed"] / s["logical"]
if share > 25:
add("medium", "RL02", "Abusive client %s still gets %.0f%% of its requests through (%d); "
"tighten the per-client limit or add a stricter policy for its pattern." % (name, share, s["allowed"]))
amp = s["attempts"] / s["logical"] if s["logical"] else 0
if amp >= 1.5:
add("medium", "RL05", "Client %s sends %.2fx attempts per logical request because it retries throttled "
"calls; retries count against the limit and add load - require Retry-After or backoff with jitter."
% (name, amp))
if per_sec and retry_sec.get(peak_sec, 0) > 0.5 * peak_rps and peak_rps >= 20:
add("medium", "RL10", "Retry herd: %d of the %d requests admitted in second %d are retries released by the same "
"Retry-After / window reset; add jitter to Retry-After (or client backoff) and prefer a token bucket or "
"sliding window so capacity is not handed out all at once." % (retry_sec[peak_sec], peak_rps, peak_sec))
cap = cfg.get("capacity_rps")
if cap:
if peak_rps > cap:
add("high", "RL03", "Admitted traffic peaks at %d req/s (second %d) over the backend capacity of %g req/s; "
"add or lower a global limit." % (peak_rps, peak_sec, cap))
if not any(p["key"] == "global" for p in pols):
per_client = [sustained_rps(p) for p in pols if p["key"] in ("client", "client+endpoint")
and not p["endpoints"] and not p["clients"]]
if per_client:
n = len(stats)
add("medium", "RL03", "No global policy: %d clients x %g req/s per client = %g req/s possible vs capacity %g."
% (n, min(per_client), n * min(per_client), cap))
policies = []
for p, ps in zip(pols, pstats):
obs = max((max_in_window(ts, p["window_s"]) for ts in ps["allowed_times"].values()), default=0)
theo = theoretical_max(p)
policies.append({"name": p["name"], "algorithm": p["algorithm"], "key": p["key"],
"endpoints": p["endpoints"], "clients": p["clients"], "exclude_clients": p["exclude_clients"], "checks": ps["checks"], "rejects": ps["rejects"],
"sustained_rps": round(sustained_rps(p), 3), "window_s": p["window_s"],
"worst_case_per_window": round(theo, 1), "observed_max_per_window": obs})
if p["algorithm"] == "fixed_window" and obs > p["limit"]:
add("medium", "RL04", "Fixed window '%s' let %d requests through in one rolling %gs window (limit %g, "
"%.1fx) across a window boundary; use sliding_counter or a token bucket if the limit protects capacity."
% (p["name"], obs, p["window_s"], p["limit"], obs / p["limit"]))
if p["key"] == "global" and ps["rejects"]:
allowed_by = defaultdict(int)
for t, cl, _ in admitted:
allowed_by[cl] += 1
total = sum(allowed_by.values()) or 1
hog = max(allowed_by, key=allowed_by.get)
victims = [c for c in ps["by_client"] if kinds.get(c) == "legit" and c != hog]
if allowed_by[hog] / total > 0.5 and victims:
add("high" if kinds.get(hog) != "legit" else "medium", "RL06",
"Noisy neighbour: %s holds %.0f%% of admitted traffic while global policy '%s' rejects %s; "
"add a per-client limit below the global one." % (hog, 100.0 * allowed_by[hog] / total,
p["name"], ", ".join(sorted(victims))))
if p["algorithm"] == "sliding_log" and p["limit"] > 1000:
add("low", "RL07", "Sliding log '%s' stores up to %d timestamps per key; prefer sliding_counter for large limits."
% (p["name"], p["limit"]))
if ps["checks"] and not ps["rejects"] and obs and theo > 10 * obs:
add("low", "RL09", "Policy '%s' never triggered; its worst case (%g per %gs) is >10x the busiest key (%d). "
"Fine as a safety net, but it will not stop a surge." % (p["name"], theo, p["window_s"], obs))
if sample:
add("info", "RL08", "Return 429 with Retry-After on every rejection (see the sample headers) and document the "
"limits so clients can pace themselves.")
order = {"high": 0, "medium": 1, "low": 2, "info": 3}
findings.sort(key=lambda f: (order[f["severity"]], f["code"]))
return {"service": cfg.get("service", "API"), "duration_s": cfg["duration_s"], "requests": len(reqs),
"admitted": len(admitted), "peak_admitted_rps": peak_rps, "peak_second": peak_sec,
"capacity_rps": cap, "clients": clients, "policies": policies, "findings": findings,
"sample_429": sample}
# ----------------------------------------------------------------------------- report
def fmt(x):
return ("%g" % x) if isinstance(x, float) else str(x)
def scope(p):
parts = []
if p["endpoints"]:
parts.append(", ".join(p["endpoints"]))
if p["clients"]:
parts.append("only " + ", ".join(p["clients"]))
if p["exclude_clients"]:
parts.append("not " + ", ".join(p["exclude_clients"]))
return "; ".join(parts) or "all"
def report(r):
L = ["# Rate limit simulation: %s" % r["service"], ""]
cap = (" vs capacity %g" % r["capacity_rps"]) if r["capacity_rps"] else ""
L.append("Simulated %gs, %d logical requests, %d admitted, peak %d req/s admitted (second %d)%s."
% (r["duration_s"], r["requests"], r["admitted"], r["peak_admitted_rps"], r["peak_second"], cap))
highs = sum(f["severity"] == "high" for f in r["findings"])
L += ["", "**Verdict: %s**" % ("%d high finding(s) - fix before shipping these limits" % highs if highs
else "policies hold - no high findings"), ""]
L += ["## Clients", "", "| Client | Kind | Requests | Attempts | Allowed | 429s | OK after retry | Failed | Failed % | Max Retry-After | Top policy |",
"|---|---|---|---|---|---|---|---|---|---|---|"]
for c in r["clients"]:
L.append("| %s | %s | %d | %d | %d | %d | %d | %d | %.1f%% | %ss | %s |" % (
c["client"], c["kind"], c["logical"], c["attempts"], c["allowed"], c["rejected"], c["delayed"], c["failed"],
c["failed_pct"], fmt(c["max_retry_after_s"]), c["top_policy"] or "-"))
L += ["", "## Policies", "", "| Policy | Algorithm | Key | Scope | Sustained | Worst case / window | Busiest key / window | Checks | Rejects |",
"|---|---|---|---|---|---|---|---|---|"]
for p in r["policies"]:
L.append("| %s | %s | %s | %s | %s req/s | %s per %ss | %d | %d | %d |" % (
p["name"], p["algorithm"], p["key"], scope(p), fmt(p["sustained_rps"]),
fmt(p["worst_case_per_window"]), fmt(p["window_s"]), p["observed_max_per_window"], p["checks"], p["rejects"]))
L += ["", "## Findings", ""]
if not r["findings"]:
L.append("- none")
for f in r["findings"]:
L.append("- **%s %s** %s" % (f["severity"].upper(), f["code"], f["message"]))
s = r["sample_429"]
if s:
L += ["", "## Sample 429 (first rejection, t=%ss, %s %s)" % (s["t"], s["client"], s["endpoint"]), "", "```",
"HTTP/1.1 429 Too Many Requests",
"Retry-After: %d" % s["retry_after"],
'RateLimit-Policy: "%s";q=%s;w=%s' % (s["policy"], fmt(float(s["quota"])), fmt(float(s["window_s"]))),
'RateLimit: "%s";r=0;t=%d' % (s["policy"], s["retry_after"]),
"Content-Type: application/problem+json", "```",
"Header names follow the IETF httpapi ratelimit-headers draft; older gateways use "
"X-RateLimit-Limit / -Remaining / -Reset instead."]
return "\n".join(L) + "\n"
# ----------------------------------------------------------------------------- self-test
def self_test():
cases = []
def check(name, cond, detail=""):
cases.append((name, bool(cond), detail))
tb = TokenBucket(rate=1, burst=3)
ok = 0
for i in range(5):
a, w, _ = tb.check("k", 0.0, 1)
if a:
tb.consume("k", 0.0, 1)
ok += 1
check("token bucket burst of 3 at t=0", ok == 3, ok)
a, w, _ = tb.check("k", 0.0, 1)
check("token bucket retry-after 1s", (not a) and abs(w - 1.0) < 1e-6, w)
a, w, _ = tb.check("k", 1.0, 1)
check("token bucket refills 1 token per second", a)
fw = FixedWindow(limit=10, window=60)
times = [59.0 + i * 0.01 for i in range(10)] + [60.0 + i * 0.01 for i in range(10)]
allowed = []
for t in times:
a, _, _ = fw.check("k", t, 1)
if a:
fw.consume("k", t, 1)
allowed.append(t)
check("fixed window boundary burst = 2x limit", max_in_window(allowed, 60) == 20, len(allowed))
a, w, _ = fw.check("k", 61.0, 1)
check("fixed window retry-after until reset", (not a) and abs(w - 59.0) < 1e-6, w)
sl = SlidingLog(limit=10, window=60)
allowed = []
for t in times:
a, _, _ = sl.check("k", t, 1)
if a:
sl.consume("k", t, 1)
allowed.append(t)
check("sliding log never exceeds limit", max_in_window(allowed, 60) == 10, len(allowed))
a, w, _ = sl.check("k", 60.5, 1)
check("sliding log retry-after = oldest + window", (not a) and abs(w - (59.0 + 60 - 60.5)) < 1e-6, w)
sc = SlidingCounter(limit=10, window=60)
allowed = []
for t in times + [90.0 + i for i in range(10)]:
a, _, _ = sc.check("k", t, 1)
if a:
sc.consume("k", t, 1)
allowed.append(t)
first = [t for t in allowed if t < 61]
later = [t for t in allowed if t >= 90]
check("sliding counter blocks the boundary burst", len(first) == 10, len(first))
check("sliding counter weights previous window (about half at mid-window)", 4 <= len(later) <= 7, len(later))
cfg = normalize({"service": "t", "duration_s": 60, "capacity_rps": 1000, "policies": [
{"name": "pc", "algorithm": "token_bucket", "key": "client", "rate": 5, "burst": 5}],
"clients": [{"name": "good", "kind": "legit", "rps": 2},
{"name": "bad", "kind": "abusive", "rps": 20, "retry": "immediate"}]})
r = analyse(cfg, generate(cfg))
by = {c["client"]: c for c in r["clients"]}
check("legit client under limit not throttled", by["good"]["rejected"] == 0, by["good"])
check("abusive client mostly throttled", by["bad"]["allowed"] <= 5 * 60 + 10, by["bad"]["allowed"])
check("immediate retries amplify attempts", by["bad"]["amplification"] > 1.5, by["bad"]["amplification"])
codes = {f["code"] for f in r["findings"]}
check("RL05 retry amplification finding", "RL05" in codes, codes)
cfg2 = normalize({"duration_s": 30, "capacity_rps": 10, "policies": [
{"name": "g", "algorithm": "fixed_window", "key": "global", "limit": 600, "window_s": 60}],
"clients": [{"name": "hog", "kind": "abusive", "rps": 30},
{"name": "app", "kind": "legit", "rps": 1, "start": 25, "duration": 5}]})
r2 = analyse(cfg2, generate(cfg2))
codes2 = {f["code"]: f["severity"] for f in r2["findings"]}
check("RL03 over capacity is high", codes2.get("RL03") == "high", codes2)
check("RL06 noisy neighbour on global", "RL06" in codes2, codes2)
check("RL01 legit client starved", codes2.get("RL01") == "high", codes2)
cfg3 = normalize({"duration_s": 10, "policies": [
{"name": "login", "algorithm": "sliding_log", "key": "client", "limit": 2, "window_s": 60, "endpoints": ["/login"]}],
"clients": [{"name": "u", "rps": 1, "endpoint": "/orders"}, {"name": "v", "rps": 1, "endpoint": "/login"}]})
r3 = analyse(cfg3, generate(cfg3))
by3 = {c["client"]: c for c in r3["clients"]}
check("endpoint-scoped policy ignores other endpoints", by3["u"]["rejected"] == 0 and by3["v"]["allowed"] == 2,
(by3["u"], by3["v"]))
check("burst pattern generates bursts", len(generate(normalize({"duration_s": 120, "policies": cfg3["policies"],
"clients": [{"name": "b", "pattern": "burst", "burst": 5, "every_s": 60}]}))) == 10)
cfg4 = normalize({"duration_s": 10, "policies": [
{"name": "default", "algorithm": "token_bucket", "key": "client", "rate": 1, "burst": 1, "exclude_clients": ["vip"]},
{"name": "vip", "algorithm": "token_bucket", "key": "client", "rate": 10, "burst": 10, "clients": ["vip"]}],
"clients": [{"name": "vip", "rps": 5}, {"name": "std", "rps": 5}]})
by4 = {c["client"]: c for c in analyse(cfg4, generate(cfg4))["clients"]}
check("client tiers: vip unthrottled, std limited", by4["vip"]["rejected"] == 0 and by4["std"]["allowed"] <= 11,
(by4["vip"], by4["std"]))
try:
normalize({"policies": [{"algorithm": "leaky"}]})
check("bad algorithm rejected", False)
except InputError:
check("bad algorithm rejected", True)
rep = report(r2)
check("report has verdict and sample 429", "Verdict" in rep and "429 Too Many Requests" in rep)
for name, ok, detail in cases:
print("%s %s%s" % ("PASS" if ok else "FAIL", name, "" if ok else " -> %r" % (detail,)))
passed = sum(1 for _, ok, _ in cases if ok)
print("%d/%d passed" % (passed, len(cases)))
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Simulate API rate limit policies against traffic.")
ap.add_argument("config", nargs="?")
ap.add_argument("--trace", help="CSV with t,client[,endpoint][,cost] instead of generated traffic")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.config:
ap.print_usage(sys.stderr)
return 2
try:
with open(a.config, encoding="utf-8") as f:
cfg = normalize(json.load(f))
reqs = read_trace(a.trace) if a.trace else generate(cfg)
if not reqs:
raise InputError("no traffic: add clients to the config or pass --trace")
r = analyse(cfg, reqs)
except (InputError, OSError, json.JSONDecodeError) as e:
print("error: %s" % e, file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="" if not a.json else "\n")
return 1 if any(f["severity"] == "high" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Source: github.com/f/prompts.chat · Contributor: f
Start from the step 2 image "Glass Igloos Glowing Under a Green and Violet Aurora in Snowy Lapland" as the exact first frame, and animate it into a calm 6-second cinematic shot. Keep the large glass igloo on the right-center with its lit door and cozy interior, the two smaller glowing igloos further back, the snow-laden spruce trees on both sides, the footprint trail in the deep snow, the wooden signpost with blank arrow boards, the colors, and the composition exactly as in the image. The green and violet aurora ripples and shifts slowly across the sky in soft curtains, and the stars twinkle. Light snow drifts gently through the air, and a small puff of powder snow slides off a spruce branch on the right and falls softly. Warm golden light glows steadily inside the igloos. The camera makes a slow, smooth push-in along the footprint trail toward the big igloo, and a few more small glowing igloos come into view far back among the trees. The shot ends calm and magical in the cold blue night. Natural, realistic motion, no people, no cuts, no text. Soft sound of a gentle wind, faint snow falling, and deep arctic silence.Source: github.com/f/prompts.chat · Contributor: f
Glass-domed igloo cabins glowing with warm golden light in deep fresh snow among heavily snow-laden spruce trees under a swirling green and violet aurora in Lapland at night, wide landscape 16:9, seen from eye level. The largest glass igloo stands on the right-center with a metal-framed dome, a lit glass door, and a cozy bed and armchairs visible inside, resting on a snowy base. Two smaller glowing glass igloos sit further back, one on the far left and one in the center, half-hidden between the trees. A trail of footprints runs through the deep snow from the bottom center of the frame toward the big igloo. A wooden signpost with three blank arrow boards stands in the snow on the right, just in front of the big igloo. Tall spruces thick with snow frame both sides, with a dark forest line behind. The upper half of the sky is filled with a vivid green and violet aurora swirling in a wide arc, over a deep blue sky full of stars. Cold blue snow against warm amber cabin light, photoreal long-exposure night travel photography, magical, peaceful, and cozy mood. No people, no text, no logos.