
Phase 1 gave you a lot of tools. This project puts them to work. You'll build notes, a command-line tool that searches a folder of markdown notes, ranks the results, and plays nicely with pipes and scripts.
It's the first of four projects that close out the programming track. Everything below was built and run for real, with one exception: pytest couldn't be installed in my test environment, so I ran the test functions with a small stand-in, as I did in Post 13.
What we're building
Point it at a folder of notes, and ask a question:
$ notes search generators --dir sample-notes
1. Python generators [python, generators] (score 19)
python/generators.md:5: # Python generators
python/generators.md:12: Generators keep memory small when a file is huge.
2. Decorators [python] (score 1)
python/decorators.md:7: It is similar to how generators wrap a loop.Each result shows the note's title, its tags, a score, and the matching lines with their line numbers, so you can jump straight to them in an editor.
The tool has three commands:
Command | What it does |
|---|---|
| Finds notes containing all the words, best matches first |
| Lists every note, with its title and tags |
| Counts notes, words, and tags |
And a few options: --dir (which folder), --tag (filter by tag), --limit (how many results), --json (machine-readable output), and -v (show which files are read).
It follows the conventions of good command-line tools, which makes it usable in scripts:
Exit code | Meaning |
|---|---|
0 | Success, and (for |
1 | Nothing matched, or an unknown tag, like |
2 | Bad usage: a missing folder, a bad option, no command |
Results go to stdout and messages go to stderr, so piping the output into another program never mixes in warnings.
The notes it reads
Notes are plain .md files, optionally starting with a small block of metadata between two --- lines:
---
title: Python generators
tags: [python, generators]
---
# Python generators
A generator yields values lazily, one at a time.
Streaming replies from a model use the same idea.Without that block, the title comes from the first # Heading, and failing that, from the file name. The sample folder I used has four notes (python/generators.md, python/decorators.md, ai/rag.md, and ideas.md) plus a .hidden/secret.md that the tool should ignore. Make your own, or point the tool at real notes.
The plan
Three small modules, each with one job:
notesearch/
├── pyproject.toml
├── README.md
├── .gitignore
├── src/notesearch/
│ ├── __init__.py
│ ├── notes.py # read files, parse front matter (no searching, no printing)
│ ├── search.py # score notes against a query (no files, no printing)
│ └── cli.py # arguments, output, exit codes (the only part that prints)
└── tests/
└── test_notesearch.pyKeeping the parts apart pays off when we test: parsing and scoring need no files or terminal at all.
The scoring is deliberately simple and easy to explain. A note matches only if every search word appears somewhere in it. Then each word earns points:
Where the word appears | Points |
|---|---|
In the title | 10 |
In a tag | 5 |
On a heading line (starts with | 3 per line |
On any other line | 1 per line |
Ties are broken by file path, so results are always in a stable order.
Along the way you'll use a good chunk of the track: generators for lazy file reading, dataclasses for the data, heapq for picking the best results, difflib for a "did you mean?" on tags, logging, pathlib, packaging with a console script, tests, and Git.
Step 0: set up the project
mkdir notesearch && cd notesearch
git init -b main
python -m venv .venv
source .venv/bin/activate # on Windows, use the activate line from Post 12
mkdir -p src/notesearch tests sample-notesCreate pyproject.toml. The important part is [project.scripts], which turns a function into a command:
[project]
name = "notesearch"
version = "0.1.0"
description = "Search and browse a folder of markdown notes from the command line"
readme = "README.md"
requires-python = ">=3.11"
dependencies = []
[project.optional-dependencies]
dev = ["pytest>=8.0"]
[project.scripts]
notes = "notesearch.cli:main"
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.pytest.ini_options]
testpaths = ["tests"]notes = "notesearch.cli:main" means: create a command called notes that runs the main function in notesearch/cli.py. Add a .gitignore containing .venv/, __pycache__/, *.egg-info/, and .pytest_cache/. Add a short README.md, and a one-line src/notesearch/__init__.py:
__version__ = "0.1.0"Commit this skeleton before writing any logic, with a message like Set up project skeleton with pyproject and README.
Step 1: reading notes
Create src/notesearch/notes.py:
import logging
from collections.abc import Iterator
from dataclasses import dataclass
from pathlib import Path
logger = logging.getLogger("notesearch")
@dataclass(frozen=True)
class Note:
path: Path
title: str
tags: tuple[str, ...]
body: tuple[str, ...] # the lines after the front matter
body_start: int # line number in the file of body[0], counting from 1
@property
def word_count(self) -> int:
return sum(len(line.split()) for line in self.body)
def parse_front_matter(lines: list[str]) -> tuple[dict[str, str], int]:
"""Read a leading block of 'key: value' lines between two '---' lines.
Returns the values and how many lines the block used (0 if there isn't one).
"""
if not lines or lines[0].strip() != "---":
return {}, 0
for end, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
values = {}
for entry in lines[1:end]:
key, separator, value = entry.partition(":")
if separator:
values[key.strip().lower()] = value.strip()
return values, end + 1
return {}, 0 # an opening '---' with no closing one: treat it as plain text
def parse_tags(raw: str) -> tuple[str, ...]:
"""Accept 'a, b' or '[a, b]', and return lowercase tags without duplicates."""
cleaned = raw.strip().strip("[]")
tags = [tag.strip().strip("'\"").lower() for tag in cleaned.split(",")]
return tuple(dict.fromkeys(tag for tag in tags if tag))
def first_heading(lines: list[str]) -> str | None:
for line in lines:
if line.startswith("# "):
return line[2:].strip()
return None
def parse_note(path: Path, text: str) -> Note:
lines = text.splitlines()
meta, used = parse_front_matter(lines)
body = lines[used:]
title = meta.get("title") or first_heading(body) or path.stem
return Note(
path=path,
title=title,
tags=parse_tags(meta.get("tags", "")),
body=tuple(body),
body_start=used + 1,
)
def iter_notes(root: Path) -> Iterator[Note]:
"""Yield every markdown note under root, one at a time, skipping hidden folders."""
for path in sorted(root.rglob("*.md")):
relative_parts = path.relative_to(root).parts
if any(part.startswith(".") for part in relative_parts):
continue
try:
text = path.read_text(encoding="utf-8", errors="replace")
except OSError as error:
logger.warning("skipping %s: %s", path, error)
continue
logger.debug("read %s", path)
yield parse_note(path, text)A few choices deserve a comment.
The front matter parser is tiny on purpose. Real front matter is YAML, and a library like PyYAML parses all of it. But we only need title and tags, and 15 lines of our own code means zero dependencies. The cost is that it understands only simple key: value lines, which is fine for notes, and worth a sentence in the README so nobody is surprised.
Malformed input doesn't crash it. An opening --- with no closing one is treated as ordinary text, and the three tag styles ([a, b], a, b, and quoted) all work. Notes written by humans are messy, so the parser forgives. I checked the edge cases directly:
parse_front_matter(["---", "title: A", "tags: [x]", "---", "body"]) # ({'title': 'A', 'tags': '[x]'}, 4)
parse_front_matter(["---", "title: A", "body"]) # ({}, 0)
parse_tags("[a, B, a]") # ('a', 'b')body_start makes line numbers honest. Search results point at lines in the file, but we've stripped the front matter off the body. So each note remembers where its body began, and body_start + offset gives the real line number.
iter_notes is a generator (Post 7). It yields one note at a time instead of building a list, so notes stats can run over a huge folder without keeping it all in memory. It skips hidden folders like .git, and if one file can't be read, it logs a warning and carries on instead of dying.
Step 2: searching and scoring
Create src/notesearch/search.py:
import heapq
from collections.abc import Iterable
from dataclasses import dataclass
from notesearch.notes import Note
TITLE_POINTS = 10
TAG_POINTS = 5
HEADING_POINTS = 3
BODY_POINTS = 1
MAX_HITS_PER_NOTE = 3
@dataclass(frozen=True)
class Hit:
line_number: int
text: str
@dataclass(frozen=True)
class Result:
note: Note
score: int
hits: tuple[Hit, ...]
def split_query(query: str) -> list[str]:
return query.lower().split()
def score_note(note: Note, terms: list[str]) -> Result | None:
"""Score a note, or return None unless it contains every search term."""
title = note.title.lower()
tags = " ".join(note.tags)
body = [line.lower() for line in note.body]
everything = " ".join([title, tags, *body])
if not terms or any(term not in everything for term in terms):
return None
score = 0
for term in terms:
if term in title:
score += TITLE_POINTS
if term in tags:
score += TAG_POINTS
for line in body:
if term in line:
score += HEADING_POINTS if line.startswith("#") else BODY_POINTS
hits = []
for offset, (original, lowered) in enumerate(zip(note.body, body)):
if any(term in lowered for term in terms):
hits.append(Hit(note.body_start + offset, original.strip()))
if len(hits) == MAX_HITS_PER_NOTE:
break
return Result(note, score, tuple(hits))
def search(notes: Iterable[Note], query: str, tag: str | None = None, limit: int = 10) -> list[Result]:
"""The best-scoring notes first, ties broken by file path."""
terms = split_query(query)
wanted = tag.lower() if tag else None
results = []
for note in notes:
if wanted and wanted not in note.tags:
continue
result = score_note(note, terms)
if result:
results.append(result)
return heapq.nsmallest(limit, results, key=lambda r: (-r.score, str(r.note.path)))The point values are named constants, so tuning the ranking later means changing a number and not hunting through the logic. score_note is a pure function: it takes a note and some words, and returns a result or None. No files, no printing, so it's trivial to test.
heapq.nsmallest picks the top results without sorting everything (Post 16). Since it finds the smallest items, the key negates the score, so the biggest score comes first, and ties fall back to the path string.
Check the scores by hand against the sample output. For generators, the "Python generators" note earns 10 (title) + 5 (tag) + 3 (the heading line mentions it) + 1 (one body line) = 19. The "Decorators" note mentions it once in its body, so it earns 1.
Step 3: a crash course in argparse
The command-line part uses argparse, which comes with Python and builds argument parsing, error messages, and --help for you. It's new in this series, so here are the pieces we use:
Positional arguments (
query) are required and identified by position. Options (--dir,-t) start with dashes and usually have defaults.type=Pathortype=intconverts the text for you, and rejects bad values with a clear message.action="store_true"makes a flag, like--json, that'sFalseunless given.nargs="+"means "one or more values", sonotes search streaming modelgivesquery == ["streaming", "model"].Subcommands, like
git commitandgit push, come fromadd_subparsers.A parent parser holds options shared by several subcommands. Without it,
--dirwould have to come before the subcommand, andnotes search x --dir ywould be an error.parse_args(argv)readssys.argvby default, but accepts a list, which is exactly what tests need.On bad usage,
argparseprints the usage line to stderr and exits with code 2.
Step 4: the command line itself
Create src/notesearch/cli.py:
import argparse
import difflib
import json
import logging
import os
import sys
from collections import Counter
from pathlib import Path
from notesearch import __version__
from notesearch.notes import Note, iter_notes
from notesearch.search import Result, search
def build_parser() -> argparse.ArgumentParser:
common = argparse.ArgumentParser(add_help=False)
common.add_argument("-d", "--dir", type=Path, default=Path("."),
help="folder of notes (default: the current folder)")
common.add_argument("-v", "--verbose", action="store_true", help="show which files are read")
parser = argparse.ArgumentParser(prog="notes", description="Search and browse a folder of markdown notes.")
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
commands = parser.add_subparsers(dest="command", required=True, metavar="command")
find = commands.add_parser("search", parents=[common], help="find notes that contain all the given words")
find.add_argument("query", nargs="+", help="words to look for (all of them must match)")
find.add_argument("-t", "--tag", help="only look at notes with this tag")
find.add_argument("-n", "--limit", type=int, default=10, help="most notes to show (default: 10)")
find.add_argument("--json", action="store_true", help="print JSON instead of text")
listing = commands.add_parser("list", parents=[common], help="list notes")
listing.add_argument("-t", "--tag", help="only notes with this tag")
listing.add_argument("--json", action="store_true", help="print JSON instead of text")
commands.add_parser("stats", parents=[common], help="count notes, words, and tags")
return parser
def relative(note: Note, root: Path) -> str:
return str(note.path.relative_to(root))
def result_to_dict(result: Result, root: Path) -> dict:
return {
"path": relative(result.note, root),
"title": result.note.title,
"tags": list(result.note.tags),
"score": result.score,
"hits": [{"line": hit.line_number, "text": hit.text} for hit in result.hits],
}
def print_results(results: list[Result], root: Path, as_json: bool) -> None:
if as_json:
print(json.dumps([result_to_dict(r, root) for r in results], indent=2))
return
for rank, result in enumerate(results, start=1):
tags = f" [{', '.join(result.note.tags)}]" if result.note.tags else ""
print(f"{rank}. {result.note.title}{tags} (score {result.score})")
for hit in result.hits:
print(f" {relative(result.note, root)}:{hit.line_number}: {hit.text}")
def report_unknown_tag(tag: str, notes: list[Note]) -> int:
known = sorted({t for note in notes for t in note.tags})
message = f"notes: no notes are tagged '{tag}'."
suggestions = difflib.get_close_matches(tag.lower(), known)
if suggestions:
message += f" Did you mean: {', '.join(suggestions)}?"
print(message, file=sys.stderr)
return 1
def run_search(args: argparse.Namespace, root: Path) -> int:
if args.limit < 1:
print("notes: error: --limit must be at least 1", file=sys.stderr)
return 2
notes = list(iter_notes(root))
if args.tag and not any(args.tag.lower() in note.tags for note in notes):
return report_unknown_tag(args.tag, notes)
results = search(notes, " ".join(args.query), tag=args.tag, limit=args.limit)
print_results(results, root, args.json)
if not results and not args.json:
print("No notes matched.", file=sys.stderr)
return 0 if results else 1
def run_list(args: argparse.Namespace, root: Path) -> int:
notes = list(iter_notes(root))
if args.tag and not any(args.tag.lower() in note.tags for note in notes):
return report_unknown_tag(args.tag, notes)
if args.tag:
notes = [note for note in notes if args.tag.lower() in note.tags]
if args.json:
rows = [{"path": relative(n, root), "title": n.title, "tags": list(n.tags)} for n in notes]
print(json.dumps(rows, indent=2))
else:
for note in notes:
tags = f" [{', '.join(note.tags)}]" if note.tags else ""
print(f"{relative(note, root)} {note.title}{tags}")
return 0
def run_stats(args: argparse.Namespace, root: Path) -> int:
note_count = 0
word_count = 0
tag_counts: Counter[str] = Counter()
for note in iter_notes(root): # one note at a time: nothing is kept in memory
note_count += 1
word_count += note.word_count
tag_counts.update(note.tags)
print(f"notes: {note_count}")
print(f"words: {word_count}")
if tag_counts:
print("tags:")
for tag, count in tag_counts.most_common():
print(f" {tag}: {count}")
return 0
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
logging.basicConfig(level=logging.DEBUG if args.verbose else logging.WARNING,
format="%(levelname)s: %(message)s")
root: Path = args.dir
if not root.is_dir():
print(f"notes: error: {root} is not a folder", file=sys.stderr)
return 2
handlers = {"search": run_search, "list": run_list, "stats": run_stats}
try:
return handlers[args.command](args, root)
except BrokenPipeError:
# The reader (for example `head`) stopped early. That's normal for a pipe, so exit quietly.
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
return 0
if __name__ == "__main__":
sys.exit(main())Several decisions here are about being a good citizen on the command line:
mainreturns an int, and the entry point exits with it. The generatednotescommand effectively runssys.exit(main()). Returning codes instead of callingsys.exitall over the place keepsmaineasy to test.Messages go to stderr, results to stdout. "No notes matched." is printed to stderr in text mode, so
notes search x | somethingreceives an empty stream, not a sentence. In JSON mode the tool prints[]and still exits with 1.The unknown-tag message uses
difflib, the "did you mean?" idea from Post 17. A typo like--tag pytongets a helpful reply instead of a silent empty result.searchandlistkeep the notes in a list, because they need two passes (check the tag, then search).statsstreams them one at a time.The
BrokenPipeErrorhandler came from a bug I hit myself. I testednotes list --json | head -8, andheadclosed the pipe after eight lines, which made Python's nextprintraiseBrokenPipeError, with a long traceback. That's not an error in your program. It's the reader saying "that's enough." The fix redirects stdout to/dev/nullso Python's exit cleanup doesn't complain, and exits quietly. After the fix,notes list --json | head -2printed two lines, nothing on stderr, and both programs exited with code 0.
Here's the help text argparse generated for us, for free (the exact layout can vary by Python version):
$ notes --help
usage: notes [-h] [--version] command ...
Search and browse a folder of markdown notes.
positional arguments:
command
search find notes that contain all the given words
list list notes
stats count notes, words, and tags
options:
-h, --help show this help message and exit
--version show program's version number and exitStep 5: tests
The tests use a fixture that builds a small notes folder in a temporary directory (Post 13), so no real files are touched. Create tests/test_notesearch.py:
import json
from pathlib import Path
import pytest
from notesearch.cli import main
from notesearch.notes import iter_notes, parse_front_matter, parse_note, parse_tags
from notesearch.search import score_note, search, split_query
def write(path: Path, text: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
@pytest.fixture
def notes_dir(tmp_path):
write(tmp_path / "python" / "generators.md",
"---\ntitle: Python generators\ntags: [python, generators]\n---\n"
"# Python generators\n\nA generator yields values lazily.\nStreaming replies use the same idea.\n")
write(tmp_path / "python" / "decorators.md",
"---\ntitle: Decorators\ntags: python\n---\n"
"A decorator wraps a function.\nIt is similar to how generators wrap a loop.\n")
write(tmp_path / "ai" / "rag.md",
"---\ntitle: RAG basics\ntags: [ai, search]\n---\n"
"Retrieval augmented generation looks up documents first.\n")
write(tmp_path / "ideas.md", "# Project ideas\n\n- A streaming chat UI\n")
write(tmp_path / ".hidden" / "secret.md", "generators streaming secret\n")
return tmp_path
# ---- parsing ----
def test_front_matter_is_parsed():
values, used = parse_front_matter(["---", "title: A", "tags: [x]", "---", "body"])
assert values == {"title": "A", "tags": "[x]"}
assert used == 4
@pytest.mark.parametrize("lines", [
["no front matter"],
["---", "title: A", "body"], # the closing marker is missing
[],
])
def test_text_without_a_complete_front_matter_block_is_plain_text(lines):
assert parse_front_matter(lines) == ({}, 0)
@pytest.mark.parametrize("raw, expected", [
("[a, B, a]", ("a", "b")),
("python, generators", ("python", "generators")),
("['x', \"y\"]", ("x", "y")),
("", ()),
])
def test_parse_tags(raw, expected):
assert parse_tags(raw) == expected
def test_title_falls_back_to_the_first_heading_then_the_file_name():
assert parse_note(Path("a/x.md"), "---\ntitle: Given\n---\n# Heading").title == "Given"
assert parse_note(Path("a/x.md"), "intro\n# The Heading\nmore").title == "The Heading"
assert parse_note(Path("a/untitled.md"), "plain text only").title == "untitled"
def test_line_numbers_count_the_front_matter():
note = parse_note(Path("x.md"), "---\ntitle: T\ntags: a\n---\nfirst\nsecond needle\n")
assert note.body_start == 5
result = score_note(note, ["needle"])
assert result is not None
assert [hit.line_number for hit in result.hits] == [6]
def test_hidden_folders_are_skipped(notes_dir):
names = [note.path.name for note in iter_notes(notes_dir)]
assert "secret.md" not in names
assert sorted(names) == ["decorators.md", "generators.md", "ideas.md", "rag.md"]
# ---- searching ----
def test_every_term_must_match():
note = parse_note(Path("x.md"), "alpha beta")
assert score_note(note, ["alpha", "beta"]) is not None
assert score_note(note, ["alpha", "gamma"]) is None
assert score_note(note, []) is None
def test_a_title_match_outranks_a_body_match(notes_dir):
results = search(iter_notes(notes_dir), "generators")
assert [r.note.title for r in results] == ["Python generators", "Decorators"]
assert results[0].score > results[1].score
def test_search_can_filter_by_tag(notes_dir):
titles = [r.note.title for r in search(iter_notes(notes_dir), "generators", tag="Python")]
assert titles == ["Python generators", "Decorators"]
assert search(iter_notes(notes_dir), "generators", tag="ai") == []
def test_limit_and_tie_order(notes_dir):
# 'streaming' appears once in ideas.md and once in generators.md: equal scores, ordered by path
both = search(iter_notes(notes_dir), "streaming")
assert [r.score for r in both] == [1, 1]
assert [r.note.path.name for r in both] == ["ideas.md", "generators.md"]
assert len(search(iter_notes(notes_dir), "streaming", limit=1)) == 1
def test_split_query_lowercases_and_splits():
assert split_query(" Generators STREAMING ") == ["generators", "streaming"]
# ---- the command line ----
def test_search_prints_the_best_note_first(notes_dir, capsys):
code = main(["search", "generators", "--dir", str(notes_dir)])
out = capsys.readouterr().out
assert code == 0
assert out.splitlines()[0].startswith("1. Python generators")
assert "python/generators.md:" in out
def test_json_output_is_valid_json(notes_dir, capsys):
code = main(["search", "retrieval", "-d", str(notes_dir), "--json"])
data = json.loads(capsys.readouterr().out)
assert code == 0
assert data[0]["title"] == "RAG basics"
assert data[0]["tags"] == ["ai", "search"]
assert data[0]["hits"][0]["line"] == 5
def test_no_results_gives_exit_code_1_and_keeps_stdout_clean(notes_dir, capsys):
code = main(["search", "quantum", "-d", str(notes_dir)])
captured = capsys.readouterr()
assert code == 1
assert captured.out == ""
assert "No notes matched" in captured.err
def test_a_missing_folder_gives_exit_code_2(tmp_path, capsys):
code = main(["search", "x", "-d", str(tmp_path / "nope")])
assert code == 2
assert "is not a folder" in capsys.readouterr().err
def test_an_unknown_tag_suggests_the_closest_one(notes_dir, capsys):
code = main(["search", "generators", "--tag", "pyton", "-d", str(notes_dir)])
assert code == 1
assert "Did you mean: python?" in capsys.readouterr().err
def test_stats_counts_notes_and_tags(notes_dir, capsys):
assert main(["stats", "-d", str(notes_dir)]) == 0
out = capsys.readouterr().out
assert "notes: 4" in out
assert " python: 2" in out
def test_a_command_is_required():
with pytest.raises(SystemExit) as excinfo:
main([])
assert excinfo.value.code == 2Count what these cover: the parser and its edge cases (including text with no closing marker), the line-number arithmetic, hidden folders, the AND rule, ranking, tag filters, tie ordering, and every exit code, with both text and JSON output. The CLI tests call main([...]) directly with a list, and read what it printed with the capsys fixture, so there's no subprocess and no real terminal.
The suite has 18 test functions, which expand to 23 test cases because of parametrize, and all 23 passed. Then came the check that matters, the one from Post 13: can the tests fail? I broke the code three ways, one at a time:
Deliberate bug | Tests that failed |
|---|---|
Line numbers off by one | 2 (the line-number test and the JSON test) |
Match any word instead of all | 1 ( |
Exit with 0 even when nothing is found | 1 (the no-results test) |
After restoring the code, all 23 passed again. (My first attempt at this check gave a confusing result, because Python had cached the old compiled version of a file whose size hadn't changed. If you ever see a "fixed" bug still failing, clear the __pycache__ folders.)
Step 6: install it and use it
Install the project in editable mode, so edits take effect immediately, along with the dev tools:
pip install -e ".[dev]"
pytest
which notes(In my offline test environment, I installed without the dev extras and used the stand-in runner for the tests. Your pytest run should end with 23 passed.) Now try it. Searching with two words requires both:
$ notes search streaming model -d sample-notes
1. RAG basics [ai, search] (score 2)
ai/rag.md:6: Chunking and embeddings decide what the model sees.
ai/rag.md:7: Streaming the answer makes the app feel fast.
2. Python generators [python, generators] (score 2)
python/generators.md:8: Streaming replies from a model use the same idea.Both notes score 2, so the tie is broken alphabetically by path. Filter by tag, and ask for JSON:
$ notes search retrieval -d sample-notes --json
[
{
"path": "ai/rag.md",
"title": "RAG basics",
"tags": ["ai", "search"],
"score": 1,
"hits": [
{"line": 5, "text": "Retrieval augmented generation looks up documents first."}
]
}
](The tool prints the arrays on separate lines with indent=2; I've compacted them here to save space.) Browse and count:
$ notes list -d sample-notes
ai/rag.md RAG basics [ai, search]
ideas.md Project ideas
python/decorators.md Decorators [python]
python/generators.md Python generators [python, generators]
$ notes stats -d sample-notes
notes: 4
words: 95
tags:
python: 2
ai: 1
search: 1
generators: 1Note that the hidden .hidden/secret.md appears in none of these. And the failure cases behave the way the table promised:
Command | What happens | Exit code |
|---|---|---|
| prints | 1 |
|
| 1 |
|
| 2 |
|
| 2 |
|
| 2 |
| usage message, | 2 |
Because the tool follows the conventions, it composes with everything else you know from the shell. Use the exit code in a script:
if notes search quantum -d sample-notes 2>/dev/null; then
echo "found"
else
echo "nothing found"
fi
# nothing foundOr feed the JSON to another program:
notes search generators -d sample-notes --json | python3 -c "import json,sys; print([r['title'] for r in json.load(sys.stdin)])"
# ['Python generators', 'Decorators']And -v shows what's happening under the hood, through the logging module, on stderr:
$ notes stats -d sample-notes -v
DEBUG: read sample-notes/ai/rag.md
DEBUG: read sample-notes/ideas.md
DEBUG: read sample-notes/python/decorators.md
DEBUG: read sample-notes/python/generators.md
notes: 4
words: 95
...Step 7: a clean history
Don't commit everything in one lump at the end. Commit as you build, with messages that say what changed (Post 22). The history I made while building this was:
c1f1bb0 Add tests for parsing, search, and the CLI
2fa7cda Add command line interface with search, list, and stats
8775e14 Add scoring and search with tag filter
302e7fc Parse markdown notes with simple front matter
3e14d32 Set up project skeleton with pyproject and READMEgit status was clean afterward, and .gitignore kept .venv/ out. Create a repository on GitHub, push it, and add the project to your ai-engineer-journey repo as a pull request, as in Post 22's mini project.
Where to take it next
Add a real search index. SQLite has a built-in full-text search engine called FTS5, and it's already in Python's
sqlite3(Post 21). It ranks results with a proper algorithm and handles prefixes. Here's a working sketch that indexes the sample notes:
import sqlite3
from pathlib import Path
from notesearch.notes import iter_notes
root = Path("sample-notes")
conn = sqlite3.connect(":memory:")
conn.execute("CREATE VIRTUAL TABLE notes USING fts5(path, title, body)")
for note in iter_notes(root):
conn.execute("INSERT INTO notes VALUES (?, ?, ?)",
(str(note.path.relative_to(root)), note.title, "\n".join(note.body)))
rows = conn.execute(
"SELECT path, title, snippet(notes, 2, '[', ']', '...', 6) "
"FROM notes WHERE notes MATCH ? ORDER BY bm25(notes)",
("generators",),
).fetchall() For generators it returned the generators note first, then the decorators note, with the matches wrapped in brackets. Searching gener* also found "generation" in the RAG note. Save the index to a file, and you can skip re-reading every note on each run.
2. Color the matches when output goes to a terminal (sys.stdout.isatty()), and keep plain text when it's piped.
3. Add notes new "Title" --tags a,b, which creates a note with front matter already filled in.
4. Read a default folder from an environment variable such as NOTES_DIR, so you don't need --dir every time.
5. Support "exact phrases" and a --regex option.
6. Add a type checker and a linter (Post 24), and run them with pre-commit (Post 22).
7. Try it on your own notes, and tune the point values until the ranking feels right. This is the real test.
8. Add semantic search. Searching by meaning, so "memory-efficient iteration" finds the generators note even though it doesn't contain those words, uses embeddings. You'll build that in Post 52, and this project is a fine place to plug it in.
A definition of done
Before you call it finished, check each of these:
Every command works with
--help, and the README shows how to install and use the tool.Errors go to stderr, results go to stdout, and exit codes are 0, 1, and 2 as promised.
The tests pass, and at least one of them fails when you break the code on purpose.
The code is formatted consistently, with no leftover debug prints.
Your history reads like a story, and the project is on GitHub.
What to take away
A command-line tool is a function with a good front door. Keep parsing, scoring, and printing apart, and the logic becomes easy to test.
argparsegives you subcommands, type conversion, errors, and--helpfor almost nothing. Return an exit code frommain.Follow the conventions: results to stdout, messages to stderr, 0 for success, 1 for "nothing found", 2 for bad usage, and a
--jsonoption for other programs.Be forgiving with messy input, and keep going when one file is unreadable.
Handle
BrokenPipeError, so| headdoesn't print a traceback.Generators let a tool handle a folder of any size, one file at a time.
A test that can't fail is worth nothing. Break the code on purpose to find out.
Previous: Post 26, Intro to C++/Rust
Next in the series: Post 28, Mini Project 2, where we build a REST API with tests, Docker, and continuous integration.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!