Python Packaging

Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

When to Use This Skill

• Creating Python libraries for distribution

• Building command-line tools with entry points

• Publishing packages to PyPI or private repositories

• Setting up Python project structure

• Creating installable packages with dependencies

• Building wheels and source distributions

• Versioning and releasing Python packages

• Creating namespace packages

• Implementing package metadata and classifiers

Core Concepts

1. Package Structure

• Source layout: src/package_name/ (recommended)

• Flat layout: package_name/ (simpler but less flexible)

• Package metadata: pyproject.toml, setup.py, or setup.cfg

• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

2. Modern Packaging Standards

• PEP 517/518: Build system requirements

• PEP 621: Metadata in pyproject.toml

• PEP 660: Editable installs

• pyproject.toml: Single source of configuration

3. Build Backends

• setuptools: Traditional, widely used

• hatchling: Modern, opinionated

• flit: Lightweight, for pure Python

• poetry: Dependency management + packaging

4. Distribution

• PyPI: Python Package Index (public)

• TestPyPI: Testing before production

• Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── src/ │ └── my_package/ │ ├── init.py │ └── module.py └── tests/ └── test_module.py

Minimal pyproject.toml

[build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" version = "0.1.0" description = "A short description" authors = [{name = "Your Name", email = "you@example.com"}] readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.28.0", ]

[project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", ]

Package Structure Patterns

Pattern 1: Source Layout (Recommended)

my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── .gitignore ├── src/ │ └── my_package/ │ ├── init.py │ ├── core.py │ ├── utils.py │ └── py.typed # For type hints ├── tests/ │ ├── init.py │ ├── test_core.py │ └── test_utils.py └── docs/ └── index.md

Advantages:

• Prevents accidentally importing from source

• Cleaner test imports

• Better isolation

pyproject.toml for source layout:

[tool.setuptools.packages.find] where = ["src"]

Pattern 2: Flat Layout

my-package/ ├── pyproject.toml ├── README.md ├── my_package/ │ ├── init.py │ └── module.py └── tests/ └── test_module.py

Simpler but:

• Can import package without installing

• Less professional for libraries

Pattern 3: Multi-Package Project

project/ ├── pyproject.toml ├── packages/ │ ├── package-a/ │ │ └── src/ │ │ └── package_a/ │ └── package-b/ │ └── src/ │ └── package_b/ └── tests/

Complete pyproject.toml Examples

Pattern 4: Full-Featured pyproject.toml

[build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta"

[project] name = "my-awesome-package" version = "1.0.0" description = "An awesome Python package" readme = "README.md" requires-python = ">=3.8" license = {text = "MIT"} authors = [ {name = "Your Name", email = "you@example.com"}, ] maintainers = [ {name = "Maintainer Name", email = "maintainer@example.com"}, ] keywords = ["example", "package", "awesome"] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ]

dependencies = [ "requests>=2.28.0,<3.0.0", "click>=8.0.0", "pydantic>=2.0.0", ]

[project.optional-dependencies] dev = [ "pytest>=7.0.0", "pytest-cov>=4.0.0", "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.0.0", ] docs = [ "sphinx>=5.0.0", "sphinx-rtd-theme>=1.0.0", ] all = [ "my-awesome-package[dev,docs]", ]

[project.urls] Homepage = "https://github.com/username/my-awesome-package" Documentation = "https://my-awesome-package.readthedocs.io" Repository = "https://github.com/username/my-awesome-package" "Bug Tracker" = "https://github.com/username/my-awesome-package/issues" Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"

[project.scripts] my-cli = "my_package.cli:main" awesome-tool = "my_package.tools:run"

[project.entry-points."my_package.plugins"] plugin1 = "my_package.plugins:plugin1"

[tool.setuptools] package-dir = {"" = "src"} zip-safe = false

[tool.setuptools.packages.find] where = ["src"] include = ["my_package*"] exclude = ["tests*"]

[tool.setuptools.package-data] my_package = ["py.typed", ".pyi", "data/.json"]

Black configuration

[tool.black] line-length = 100 target-version = ["py38", "py39", "py310", "py311"] include = '.pyi?$'

Ruff configuration

[tool.ruff] line-length = 100 target-version = "py38"

[tool.ruff.lint] select = ["E", "F", "I", "N", "W", "UP"]

MyPy configuration

[tool.mypy] python_version = "3.8" warn_return_any = true warn_unused_configs = true disallow_untyped_defs = true

Pytest configuration

[tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] addopts = "-v --cov=my_package --cov-report=term-missing"

Coverage configuration

[tool.coverage.run] source = ["src"] omit = ["/tests/"]

[tool.coverage.report] exclude_lines = [ "pragma: no cover", "def repr", "raise AssertionError", "raise NotImplementedError", ]

Pattern 5: Dynamic Versioning

[build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" dynamic = ["version"] description = "Package with dynamic version"

[tool.setuptools.dynamic] version = {attr = "my_package.version"}

Or use setuptools-scm for git-based versioning

[tool.setuptools_scm] write_to = "src/my_package/_version.py"

In init.py:

src/my_package/init.py

version = "1.0.0"

Or with setuptools-scm

from importlib.metadata import version version = version("my-package")

Command-Line Interface (CLI) Patterns

Pattern 6: CLI with Click

src/my_package/cli.py

import click

@click.group() @click.version_option() def cli(): """My awesome CLI tool.""" pass

@cli.command() @click.argument("name") @click.option("--greeting", default="Hello", help="Greeting to use") def greet(name: str, greeting: str): """Greet someone.""" click.echo(f"{greeting}, {name}!")

@cli.command() @click.option("--count", default=1, help="Number of times to repeat") def repeat(count: int): """Repeat a message.""" for i in range(count): click.echo(f"Message {i + 1}")

def main(): """Entry point for CLI.""" cli()

if name == "main": main()

Register in pyproject.toml:

[project.scripts] my-tool = "my_package.cli:main"

Usage:

pip install -e . my-tool greet World my-tool greet Alice --greeting="Hi" my-tool repeat --count=3

Pattern 7: CLI with argparse

src/my_package/cli.py

import argparse import sys

def main(): """Main CLI entry point.""" parser = argparse.ArgumentParser( description="My awesome tool", prog="my-tool" )

parser.add_argument(
    "--version",
    action="version",
    version="%(prog)s 1.0.0"
)

subparsers = parser.add_subparsers(dest="command", help="Commands")

# Add subcommand
process_parser = subparsers.add_parser("process", help="Process data")
process_parser.add_argument("input_file", help="Input file path")
process_parser.add_argument(
    "--output", "-o",
    default="output.txt",
    help="Output file path"
)

args = parser.parse_args()

if args.command == "process":
    process_data(args.input_file, args.output)
else:
    parser.print_help()
    sys.exit(1)

def process_data(input_file: str, output_file: str): """Process data from input to output.""" print(f"Processing {input_file} -> {output_file}")

if name == "main": main()

Building and Publishing

Pattern 8: Build Package Locally

Install build tools

pip install build twine

Build distribution

python -m build

This creates:

dist/

my-package-1.0.0.tar.gz (source distribution)

my_package-1.0.0-py3-none-any.whl (wheel)

Check the distribution

twine check dist/*

Pattern 9: Publishing to PyPI

Install publishing tools

pip install twine

Test on TestPyPI first

twine upload --repository testpypi dist/*

Install from TestPyPI to test

pip install --index-url https://test.pypi.org/simple/ my-package

If all good, publish to PyPI

twine upload dist/*

Using API tokens (recommended):

Create ~/.pypirc

[distutils] index-servers = pypi testpypi

[pypi] username = token password = pypi-...your-token...

[testpypi] username = token password = pypi-...your-test-token...

Pattern 10: Automated Publishing with GitHub Actions

.github/workflows/publish.yml

name: Publish to PyPI

on: release: types: [created]

jobs: publish: runs-on: ubuntu-latest

steps:
  - uses: actions/checkout@v3

  - name: Set up Python
    uses: actions/setup-python@v4
    with:
      python-version: "3.11"

  - name: Install dependencies
    run: |
      pip install build twine

  - name: Build package
    run: python -m build

  - name: Check package
    run: twine check dist/*

  - name: Publish to PyPI
    env:
      TWINE_USERNAME: __token__
      TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
    run: twine upload dist/*

Advanced Patterns

Pattern 11: Including Data Files

[tool.setuptools.package-data] my_package = [ "data/.json", "templates/.html", "static/css/*.css", "py.typed", ]

Accessing data files:

src/my_package/loader.py

from importlib.resources import files import json

def load_config(): """Load configuration from package data.""" config_file = files("my_package").joinpath("data/config.json") with config_file.open() as f: return json.load(f)

Python 3.9+

from importlib.resources import files

data = files("my_package").joinpath("data/file.txt").read_text()

Pattern 12: Namespace Packages

For large projects split across multiple repositories:

Package 1: company-core

company/ └── core/ ├── init.py └── models.py

Package 2: company-api

company/ └── api/ ├── init.py └── routes.py

Do NOT include init.py in the namespace directory (company/):

company-core/pyproject.toml

[project] name = "company-core"

[tool.setuptools.packages.find] where = ["."] include = ["company.core*"]

company-api/pyproject.toml

[project] name = "company-api"

[tool.setuptools.packages.find] where = ["."] include = ["company.api*"]

Usage:

Both packages can be imported under same namespace

from company.core import models from company.api import routes

Pattern 13: C Extensions

[build-system] requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"] build-backend = "setuptools.build_meta"

[tool.setuptools] ext-modules = [ {name = "my_package.fast_module", sources = ["src/fast_module.c"]}, ]

Or with setup.py:

setup.py

from setuptools import setup, Extension

setup( ext_modules=[ Extension( "my_package.fast_module", sources=["src/fast_module.c"], include_dirs=["src/include"], ) ] )

Version Management

Pattern 14: Semantic Versioning

src/my_package/init.py

version = "1.2.3"

Semantic versioning: MAJOR.MINOR.PATCH

MAJOR: Breaking changes

MINOR: New features (backward compatible)

PATCH: Bug fixes

Version constraints in dependencies:

dependencies = [ "requests>=2.28.0,<3.0.0", # Compatible range "click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0) "pydantic>=2.0", # Minimum version "numpy==1.24.3", # Exact version (avoid if possible) ]

Pattern 15: Git-Based Versioning

[build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" dynamic = ["version"]

[tool.setuptools_scm] write_to = "src/my_package/_version.py" version_scheme = "post-release" local_scheme = "dirty-tag"

Creates versions like:

• 1.0.0 (from git tag)

• 1.0.1.dev3+g1234567 (3 commits after tag)

Testing Installation

Pattern 16: Editable Install

Install in development mode

pip install -e .

With optional dependencies

pip install -e ".[dev]" pip install -e ".[dev,docs]"

Now changes to source code are immediately reflected

Pattern 17: Testing in Isolated Environment

Create virtual environment

python -m venv test-env source test-env/bin/activate # Linux/Mac

test-env\Scripts\activate # Windows

Install package

pip install dist/my_package-1.0.0-py3-none-any.whl

Test it works

python -c "import my_package; print(my_package.version)"

Test CLI

my-tool --help

Cleanup

deactivate rm -rf test-env

Documentation

Pattern 18: README.md Template

My Package

Brief description of your package.

Installation

pip install my-package

Quick Start

from my_package import something

result = something.do_stuff()

Features

- Feature 1

- Feature 2

- Feature 3

Documentation

Full documentation: https://my-package.readthedocs.io

Development

git clone https://github.com/username/my-package.git
cd my-package
pip install -e ".[dev]"
pytest

License

MIT

## Common Patterns

### Pattern 19: Multi-Architecture Wheels

```yaml
# .github/workflows/wheels.yml
name: Build wheels

on: [push, pull_request]

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]

    steps:
      - uses: actions/checkout@v3

      - name: Build wheels
        uses: pypa/cibuildwheel@v2.16.2

      - uses: actions/upload-artifact@v3
        with:
          path: ./wheelhouse/*.whl

Pattern 20: Private Package Index

# Install from private index
pip install my-package --index-url https://private.pypi.org/simple/

# Or add to pip.conf
[global]
index-url = https://private.pypi.org/simple/
extra-index-url = https://pypi.org/simple/

# Upload to private index
twine upload --repository-url https://private.pypi.org/ dist/*

File Templates

.gitignore for Python Packages

# Build artifacts
build/
dist/
*.egg-info/
*.egg
.eggs/

# Python
__pycache__/
*.py[cod]
*$py.class
*.so

# Virtual environments
venv/
env/
ENV/

# IDE
.vscode/
.idea/
*.swp

# Testing
.pytest_cache/
.coverage
htmlcov/

# Distribution
*.whl
*.tar.gz

MANIFEST.in

# MANIFEST.in
include README.md
include LICENSE
include pyproject.toml

recursive-include src/my_package/data *.json
recursive-include src/my_package/templates *.html
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

Checklist for Publishing

-  Code is tested (pytest passing)

-  Documentation is complete (README, docstrings)

-  Version number updated

-  CHANGELOG.md updated

-  License file included

-  pyproject.toml is complete

-  Package builds without errors

-  Installation tested in clean environment

-  CLI tools work (if applicable)

-  PyPI metadata is correct (classifiers, keywords)

-  GitHub repository linked

-  Tested on TestPyPI first

-  Git tag created for release

Resources

- Python Packaging Guide: https://packaging.python.org/

- PyPI: https://pypi.org/

- TestPyPI: https://test.pypi.org/

- setuptools documentation: https://setuptools.pypa.io/

- build: https://pypa-build.readthedocs.io/

- twine: https://twine.readthedocs.io/

Best Practices Summary

- Use src/ layout for cleaner package structure

- Use pyproject.toml for modern packaging

- Pin build dependencies in build-system.requires

- Version appropriately with semantic versioning

- Include all metadata (classifiers, URLs, etc.)

- Test installation in clean environments

- Use TestPyPI before publishing to PyPI

- Document thoroughly with README and docstrings

- Include LICENSE file

- Automate publishing with CI/CD