{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"testing-with-pytest-fixtures-fake-models-and-properties-edvab","url":"https://zyvop.com/testing-with-pytest-fixtures-fake-models-and-properties-edvab","title":"Testing with pytest: Fixtures, Fake Models, and Properties","subtitle":"Learn to write fast, reliable tests with pytest, swap real AI models for fakes so tests cost nothing, and let Hypothesis hunt for the weird inputs that break your code.","tldr":"How do you test code that depends on an unpredictable AI model? This post covers pytest from the first test to fixtures and parametrize, shows how to use fake models and mocks, and introduces property-based testing, then builds a full test suite for a small package.","keywords":["Testing","Python","pytest","AI Engineer","Mocking","AI","Zero to Top AI Engineer (2026): Full Series Plan"],"entities":["Pradeep Kumar","Testing","Python","pytest","AI Engineer","Mocking","AI","Zero to Top AI Engineer (2026): Full Series Plan","ZyVOP"],"keyTakeaways":["Here's a feeling you'll know soon enough.","You change one small thing in your code, and you're not sure what else you just broke.","So you run the program by hand, try a couple of inputs, and hope."],"headings":["Tests are not evals","Your first test","When a test fails","What makes a good test","Testing that an error is raised","Many cases, one test: parametrize","Fixtures: reusable setup","monkeypatch: changing things temporarily","Testing code that calls a model","When you can't pass a fake in: mock and patch","Don't overdo it","Property-based testing: let the computer find the weird inputs","Running tests well","Mini project: a test suite for textkit","Make it yours","Practice","What to take away"],"outboundLinks":[],"contentText":"Here's a feeling you'll know soon enough. You change one small thing in your code, and you're not sure what else you just broke. So you run the program by hand, try a couple of inputs, and hope. Tests replace the hoping. A test is a small piece of code that checks another piece of code, and you can run hundreds of them in a second. This post is core, and it matters even more for AI work than for ordinary programming, because AI apps have a part that's unpredictable (the model) surrounded by a lot of code that isn't. That surrounding code is what we'll test. Tests are not evals Before we start, one distinction that will save you confusion later. Tests check your code. Given this input, does the function return that output? They're fast, free, and always give the same answer. Evals measure how good the model's answers are. They call a real model, the results vary, and they take real time and money. That's Post 60. You need both. This post is about the first kind. And the trick for testing code that talks to a model is to replace the model with a fake, which is the heart of this post. Your first test We'll use the textkit package from Post 12. Install pytest in your project's environment: pip install pytestIf you set up the dev extras in Post 12, pip install -e \".[dev]\" does it for you. With uv, it's uv add --dev pytest. Create a file called tests/test_counting.py: from textkit.counting import count_words, top_words def test_counts_repeated_words(): assert count_words(\"the cat the dog\") == {\"the\": 2, \"cat\": 1, \"dog\": 1} def test_ignores_case_and_punctuation(): assert count_words(\"The cat. the CAT,\") == {\"the\": 2, \"cat\": 2} def test_top_words_returns_at_most_n(): result = top_words(\"a b c d e f\", n=3) assert len(result) == 3 def test_top_words_orders_by_count(): result = top_words(\"x y y z z z\", n=2) assert result == [(\"z\", 3), (\"y\", 2)]Now run: pytestYou'll see something like this (your version numbers and timing will differ): ============================= test session starts ============================== collected 4 items tests/test_counting.py .... [100%] ============================== 4 passed in 0.02s ===============================Each dot is a passing test. That's the whole system. Two rules make pytest find your tests: Files are named test_something.py. Test functions are named test_something. And a test is just a function with a plain assert in it. If the assert is true, the test passes. If it's false, the test fails. When a test fails Say we break a test on purpose by changing the expected length to 4. Here's the kind of message pytest gives you: def test_top_words_returns_at_most_n(): result = top_words(\"a b c d e f\", n=3) > assert len(result) == 4 E assert 3 == 4 E + where 3 = len([('a', 1), ('b', 1), ('c', 1)]) tests/test_counting.py:14: AssertionErrorNotice it doesn't just say \"failed.\" It shows the line, the values on both sides, and where the number came from. That's a big part of why people like pytest. You rarely need extra print statements to understand a failure. One practical note. The tests import textkit, which works because you installed the package in editable mode in Post 12. If you'd rather not, you can add this to pyproject.toml so pytest finds your code in src/: [tool.pytest.ini_options] testpaths = [\"tests\"] pythonpath = [\"src\"]What makes a good test A few habits, learned the hard way by lots of programmers: One idea per test. If a test fails, the name should tell you what broke. test_ignores_case_and_punctuation is much better than test_stuff. Arrange, act, assert. Set things up, do the thing, check the result. Keeping those three steps visible makes tests easy to read. Test behavior, not insides. Check what a function returns or does, not how it does it. Then you can rewrite the inside and the tests still pass. Cover the edges. Empty input, a single item, very long input, odd characters, zero, negative numbers. Bugs hide at the edges. Keep them fast and independent. A test shouldn't rely on another test running first. Every time you find a bug, write a test that fails because of it, then fix the bug. It's the \"lock it in\" step from Post 11, and it means the same bug can never sneak back unnoticed. Testing that an error is raised Sometimes the correct behavior is to fail. Remember set_temperature from Post 11? Here's how you check it raises an error: import pytest def set_temperature(value): if not 0 <= value <= 2: raise ValueError(\"temperature must be between 0 and 2\") return value def test_temperature_rejects_out_of_range(): with pytest.raises(ValueError, match=\"temperature\"): set_temperature(5)pytest.raises is a context manager (Post 8). The test passes only if the code inside the with block raises that error. The optional match checks the message too. If no error happens, the test fails. Many cases, one test: parametrize You often want to run the same check on lots of inputs. Instead of copying the test, use a decorator with arguments (Post 8 again): import pytest from textkit.counting import count_words @pytest.mark.parametrize( \"text, expected\", [ (\"\", {}), (\"hello\", {\"hello\": 1}), (\"A a\", {\"a\": 2}), ], ) def test_count_words_edge_cases(text, expected): assert count_words(text) == expectedPytest runs this function three times, once per row, and reports each one separately. Adding a new case is one new line. Fixtures: reusable setup A lot of tests need the same starting point: a file to read, a configured object, a fake service. A fixture is a function that builds that thing, and pytest hands it to any test that asks for it by name. Remember the text pipeline from Post 7? To test it, we need a file. Pytest has a built-in fixture called tmp_path that gives you a fresh temporary folder, and we can build our own fixture on top of it: import pytest from pipeline import keep_real_text, read_lines @pytest.fixture def temp_notes(tmp_path): path = tmp_path / \"notes.txt\" path.write_text(\"line one is long enough to keep\\nshort\\n\") return path def test_pipeline_keeps_only_long_lines(temp_notes): lines = list(keep_real_text(read_lines(temp_notes))) assert lines == [\"line one is long enough to keep\"]See how it works. The test has a parameter called temp_notes. Pytest notices that name, finds the fixture with the same name, runs it, and passes in the result. Each test gets a fresh one, and the temporary folder is cleaned up afterward. (One heads-up. If your pipeline.py from Post 7 has code at the bottom that runs the pipeline, importing it in a test will run that code too. That's exactly what the if __name__ == \"__main__\": guard from Post 12 is for. Wrap the bottom part in it.) If several test files need the same fixture, put it in a file called conftest.py in the tests/ folder. Pytest picks it up automatically, with no import needed. monkeypatch: changing things temporarily Another built-in fixture, monkeypatch, lets a test change something and automatically puts it back afterward. It's perfect for environment variables, like the API key from Post 12: import os import pytest def get_api_key(): key = os.environ.get(\"MODEL_API_KEY\") if key is None: raise RuntimeError(\"Set the MODEL_API_KEY environment variable first\") return key def test_missing_key_raises(monkeypatch): monkeypatch.delenv(\"MODEL_API_KEY\", raising=False) with pytest.raises(RuntimeError): get_api_key() def test_key_is_read(monkeypatch): monkeypatch.setenv(\"MODEL_API_KEY\", \"test-key\") assert get_api_key() == \"test-key\"Both tests work no matter what's set on your computer, and neither leaves a mess behind. Testing code that calls a model Now the part you came for. Suppose you have a function that asks a model for JSON and retries if the reply isn't valid. Testing it against a real model would be slow, cost money, need an API key, and give a different answer each time. You can't test logic that way. The fix is to make the model something you can swap out, which is exactly what we set up in Post 9 with composition and protocols. Here's the function (add it as src/textkit/replies.py): import json def get_json_reply(model, prompt, max_attempts=3): \"\"\"Ask the model for JSON. Retry if the reply isn't valid JSON.\"\"\" for _ in range(max_attempts): reply = model.generate(prompt) try: return json.loads(reply) except json.JSONDecodeError: prompt = prompt + \"\\nYour last reply was not valid JSON. Reply with JSON only.\" raise ValueError(\"Model never returned valid JSON\")It takes model as a parameter, and all it needs is something with a generate method. So in tests, we hand it a fake: a tiny class that returns replies we chose in advance. class FakeModel: \"\"\"A stand-in for a real model that returns canned replies.\"\"\" def __init__(self, replies): self.replies = list(replies) self.calls = [] def generate(self, prompt): self.calls.append(prompt) return self.replies.pop(0)It also records every prompt it received in self.calls, so tests can check what we sent. Now the tests: import pytest from textkit.replies import get_json_reply def test_returns_parsed_json(): model = FakeModel(['{\"answer\": 42}']) assert get_json_reply(model, \"Give me JSON\") == {\"answer\": 42} def test_retries_after_bad_json(): model = FakeModel([\"not json\", '{\"answer\": 42}']) result = get_json_reply(model, \"Give me JSON\") assert result == {\"answer\": 42} assert len(model.calls) == 2 assert \"not valid JSON\" in model.calls[1] def test_gives_up_after_max_attempts(): model = FakeModel([\"bad\", \"bad\", \"bad\"]) with pytest.raises(ValueError, match=\"valid JSON\"): get_json_reply(model, \"Give me JSON\")These tests run instantly, cost nothing, and are the same every time. And they check real things: that good JSON is parsed, that a bad reply triggers a retry, that the retry prompt includes the correction message, and that we give up eventually instead of looping forever. When you can't pass a fake in: mock and patch Sometimes the model call is buried inside code you can't easily change. Python's built-in unittest.mock library has tools for that, and they work fine inside pytest. Mock makes an object that accepts any call and remembers it: from unittest.mock import Mock def test_works_with_a_mock_object(): model = Mock() model.generate.return_value = '{\"answer\": 1}' assert get_json_reply(model, \"x\") == {\"answer\": 1} model.generate.assert_called_once_with(\"x\")side_effect lets a mock fail or change its answer from call to call. This is great for testing retries without waiting around: def call_with_retries(func, attempts=3): for attempt in range(1, attempts + 1): try: return func() except TimeoutError: if attempt == attempts: raise def test_retries_then_succeeds(): flaky = Mock(side_effect=[TimeoutError(\"slow\"), \"ok\"]) assert call_with_retries(flaky) == \"ok\" assert flaky.call_count == 2patch temporarily replaces a real function by name. Imagine my_app.py has a function call_api that hits the network, and answer that uses it: # my_app.py def call_api(prompt): ... # the real network call def answer(question): return call_api(question).strip()from unittest.mock import patch import my_app def test_answer_strips_whitespace(): with patch(\"my_app.call_api\", return_value=\" hi \"): assert my_app.answer(\"q\") == \"hi\"The one rule people forget: patch the name where it's used, not where it was defined. Here answer looks up call_api inside my_app, so that's the name we replace. Pytest's monkeypatch.setattr(my_app, \"call_api\", lambda prompt: \" hi \") does the same job if you prefer it. Don't overdo it Mocks are powerful, and it's easy to go too far. A few cautions: Fake only at the edges: the model, the network, the clock. Don't mock your own internal functions, or you'll end up testing the mocks instead of your code. Fakes can drift. If the real model API changes and your fake doesn't, your tests pass while your app breaks. Keep a small number of tests that call the real thing, mark them, and run them only occasionally: [tool.pytest.ini_options] markers = [\"integration: tests that call a real model API (slow, cost money)\"]@pytest.mark.integration def test_real_model_returns_json(): ...Then pytest -m \"not integration\" runs everything except those, and pytest -m integration runs only them. Tools that record and replay real API responses also exist, if you want realistic data without hitting the network each time. Property-based testing: let the computer find the weird inputs Ordinary tests check examples you thought of. The trouble is that bugs live in the examples you didn't think of. Property-based testing flips it around. You describe a rule that should be true for any input, and a tool generates hundreds of inputs to try to break it. The standard library for this in Python is Hypothesis. Install it with pip install hypothesis. Here are two properties of our word counter: from hypothesis import given from hypothesis import strategies as st from textkit.counting import count_words, top_words @given(st.text(), st.integers(min_value=0, max_value=10)) def test_top_words_is_limited_and_sorted(text, n): result = top_words(text, n) assert len(result) <= n counts = [count for _, count in result] assert counts == sorted(counts, reverse=True) @given(st.text()) def test_adding_a_word_adds_one_to_its_count(text): before = count_words(text).get(\"zzz\", 0) after = count_words(text + \" zzz\")[\"zzz\"] assert after == before + 1@given(...) tells Hypothesis what kinds of inputs to create: any text, any whole number between 0 and 10. It then runs your test many times with different values, including nasty ones like empty strings and odd Unicode characters. Notice we never said what the output should be for any particular input. We said what must always hold: the result is never longer than n, the counts are sorted from biggest to smallest, and adding a word once raises its count by exactly one. When Hypothesis finds a failing input, it doesn't just show you the mess it started with. It shrinks it to the smallest failing example, which is usually much easier to understand. A function that divides by len(numbers), for example, would be caught almost instantly with the empty list []. Properties are harder to write than examples, so don't use them for everything. They shine for code with clear rules: parsers, formatters, sorting, batching, and anything where you can say \"convert it and convert it back, and you should get the original.\" Running tests well A few options you'll use constantly: Command What it does pytest Run everything pytest -v Show each test name pytest -x Stop at the first failure pytest -k counts Run only tests whose name contains \"counts\" pytest tests/test_counting.py::test_top_words_orders_by_count Run one specific test pytest --lf Re-run only the tests that failed last time pytest -s Show print output, which pytest hides by default pytest --pdb Drop into the debugger from Post 11 when a test fails Later, in Post 28, we'll set up automatic runs so your tests execute every time you push code. A test suite that only runs when you remember is a test suite that stops running. One last warning. You may hear about \"test coverage,\" a percentage showing how many lines your tests ran. It's a handy way to spot code that has no tests at all. But a high number doesn't mean good tests. You can run every line and check nothing. Aim for tests that would actually catch a mistake. Mini project: a test suite for textkit Let's add a real test suite to the textkit package from Post 12. Step 1. Add the replies.py module shown above to src/textkit/. Step 2. Update the project settings in pyproject.toml: [project.optional-dependencies] dev = [\"pytest>=8.0\", \"hypothesis>=6.0\"] [tool.pytest.ini_options] testpaths = [\"tests\"] markers = [\"integration: tests that call a real model API (slow, cost money)\"]Step 3. Install everything: pip install -e \".[dev]\"Step 4. Create three test files in tests/: test_counting.py: the four example tests from earlier, plus the parametrized edge-case test. test_replies.py: the FakeModel class, the three tests that use it, and the Mock test. test_properties.py: the two Hypothesis tests. Step 5. Run pytest. You should see all 13 tests pass: tests/test_counting.py ....... [ 53%] tests/test_properties.py .. [ 69%] tests/test_replies.py .... [100%] ============================== 13 passed in 0.31s ==============================(The seven in the first file are four examples plus three parametrized cases.) Now the most important step: try to break it. Open counting.py and make a deliberate mistake, such as removing .lower(), flipping reverse=True to reverse=False, or changing ranked[:n] to ranked[:n + 1]. Run the tests and watch which ones catch it. A test that never fails isn't protecting you. Then put the code back. Make it yours Test the command line tool. Using monkeypatch.setattr(\"sys.argv\", [\"textkit\", \"the\", \"cat\", \"the\"]) and the capsys fixture, call main() and check that the printed output contains the: 2. Add an integration test, marked with @pytest.mark.integration, that would call a real model. Confirm pytest -m \"not integration\" skips it. Install pytest-cov and run pytest --cov=textkit. Look at which lines have no tests. Write a fake model that raises an error on the first call, and test how get_json_reply should behave. Practice About 20 minutes of real effort on each before asking for help. Write five tests for safe_int from Post 11, using parametrize for the different inputs. Write a test for set_temperature that uses pytest.raises with a match, and add a second test for a value on the boundary (exactly 0 and exactly 2). Write a fixture that creates a temporary file, and use it to test read_lines from Post 7. Take an earlier project that retries an API call. Replace the real call with a Mock(side_effect=[...]) and test the retry logic without waiting. Write a property-based test for sorting: the result is in order, and it has the same items as the input. Break a function on purpose, confirm a test fails with a readable message, then fix it. Then ask an AI tutor to review your tests, and don't let it write them for you. Tests you didn't think through are the ones that give false comfort. What to take away Tests check your code and give the same answer every time. Evals check the model's quality, and come later. Name files test_*.py and functions test_*, and use plain assert. pytest.raises checks for errors, and parametrize runs one test on many inputs. Fixtures provide reusable setup, and tmp_path and monkeypatch are two built-in ones you'll use a lot. To test code that uses a model, swap in a fake. Pass it in as a parameter, or use Mock and patch when you can't. Patch where a name is used, fake only at the edges, and keep a few real integration tests. Property-based tests describe rules that must always hold, and let Hypothesis hunt for the inputs that break them. A test is only useful if it can fail, so try breaking your code on purpose. Previous: Post 12, Packaging & Environments Next in the series: Post 14, Concurrency, where we cover threads, processes, asyncio, and why LLM apps are so often async.","contentHash":"sha256:789b3c470a34acdaef8441ca9c2880626f1839391a86fb8ad2cafaabf1a323cd","authorName":"Pradeep Kumar","authorUrl":"https://zyvop.com/author/pradeep","authorSameAs":[],"category":"AI","tags":["Testing","Python","pytest","AI Engineer","Mocking"],"audience":"Software engineers and developers building applications with AI","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":12,"wordCount":2970,"faqs":null,"primaryTopic":"AI","publishedAt":"2026-10-22T08:22:40.260Z","updatedAt":"2026-10-22T08:22:40.260Z","canonicalUrl":"https://zyvop.com/testing-with-pytest-fixtures-fake-models-and-properties-edvab"}