5.1 File System & OS Automation

5.1 File System & OS Automation

Python can automate repetitive tasks involving files, folders, and directories. Instead of manually creating folders, moving files, renaming hundreds of documents, or searching through directories, Python can perform these operations programmatically.

Three important modules for file-system automation are:

1. Why Automate File-System Tasks?

File-system automation is useful whenever the same operation must be performed repeatedly.

• Rename hundreds of files.
• Organize downloads into folders.
• Move files according to their extensions.
• Copy backup files automatically.
• Find files matching a particular pattern.
• Traverse nested directories.
• Create project folders automatically.
• Remove temporary files.
• Generate daily or monthly folders.
• Prepare files for data-processing pipelines.

2. The os Module

The os module provides functions for interacting with the operating system.

import os

It can be used to:

• Get the current working directory.
• Change directories.
• List directory contents.
• Create directories.
• Rename files.
• Remove files.
• Walk through directory trees.
• Read environment variables.

3. Getting the Current Working Directory

Use os.getcwd() to find the current working directory.

import os

current_directory = os.getcwd()

print(current_directory)

The returned path depends on where the Python program is running.

4. Changing the Working Directory

Use os.chdir() to change the current working directory.

import os

os.chdir("C:/Projects")

print(os.getcwd())

5. Listing Files and Folders

Use os.listdir() to list the contents of a directory.

import os

items = os.listdir("data")

for item in items:
    print(item)

The result contains names of both files and directories.

6. Checking Whether a Path Is a File or Directory

import os

path = "data/report.csv"

if os.path.isfile(path):
    print("It is a file.")

if os.path.isdir(path):
    print("It is a directory.")

7. Working with os.path

The os.path module provides utilities for working with file-system paths.

Check Whether a Path Exists

import os

path = "data/report.csv"

if os.path.exists(path):
    print("Path exists.")

Join Paths

path = os.path.join(
    "data",
    "reports",
    "sales.csv"
)

print(path)

Get File Name

path = "data/reports/sales.csv"

print(os.path.basename(path))

Get Directory Name

print(os.path.dirname(path))

8. The pathlib Module

pathlib provides an object-oriented approach to working with file-system paths.

from pathlib import Path

A path can be represented as a Path object:

from pathlib import Path

path = Path("data")

print(path)

9. Getting the Current Directory with pathlib

from pathlib import Path

current = Path.cwd()

print(current)

The equivalent of a home directory can be obtained with:

home = Path.home()

print(home)

10. Building Paths with /

One of the convenient features of pathlib is path composition using the / operator.

from pathlib import Path

data_folder = Path("data")

file_path = data_folder / "reports" / "sales.csv"

print(file_path)

This is more readable than repeatedly concatenating strings.

11. Checking Whether a Path Exists

from pathlib import Path

path = Path("data/report.csv")

if path.exists():
    print("Path exists.")

You can also specifically test for a file or directory:

if path.is_file():
    print("File")

if path.is_dir():
    print("Directory")

12. Creating Directories

Use mkdir() to create a directory.

from pathlib import Path

folder = Path("reports")

folder.mkdir()

If parent directories may not exist, use:

folder = Path(
    "data/2026/reports"
)

folder.mkdir(
    parents=True,
    exist_ok=True
)

13. Traversing a Directory with iterdir()

from pathlib import Path

folder = Path("data")

for item in folder.iterdir():
    print(item)

Each item returned is a Path object.

You can check its type:

for item in folder.iterdir():

    if item.is_file():
        print("FILE:", item)

    elif item.is_dir():
        print("DIR :", item)

14. Finding Files with glob()

glob() can search for files matching a pattern.

from pathlib import Path

folder = Path("data")

for file in folder.glob("*.csv"):
    print(file)

This finds CSV files directly inside the specified directory.

Find Excel Files

for file in folder.glob("*.xlsx"):
    print(file)

15. Recursive Directory Search with rglob()

Use rglob() when you want to search through nested directories.

from pathlib import Path

folder = Path("data")

for file in folder.rglob("*.csv"):
    print(file)

16. Getting File Information

from pathlib import Path

file = Path(
    "reports/annual_report.pdf"
)

print(file.name)
print(file.stem)
print(file.suffix)
print(file.parent)

17. Creating and Writing Text Files

pathlib provides convenient methods for simple text file operations.

from pathlib import Path

file = Path("notes.txt")

file.write_text(
    "Python file automation"
)

Read the contents:

content = file.read_text()

print(content)

18. Renaming Files with pathlib

from pathlib import Path

old_file = Path(
    "reports/old_report.txt"
)

new_file = Path(
    "reports/final_report.txt"
)

old_file.rename(new_file)

The file is renamed to the new path.

19. Bulk File Renaming

One of the most useful automation tasks is renaming many files according to a consistent rule.

Suppose a folder contains:

report_001.txt
report_002.txt
report_003.txt

We want to rename them to:

document_001.txt
document_002.txt
document_003.txt

from pathlib import Path

folder = Path("reports")

for file in folder.glob("report_*.txt"):

    new_name = file.name.replace(
        "report_",
        "document_",
        1
    )

    file.rename(
        file.with_name(new_name)
    )

20. Sequential File Renaming

Files can also be renamed using sequential numbering.

from pathlib import Path

folder = Path("images")

files = sorted(
    folder.glob("*.jpg")
)

for number, file in enumerate(
    files,
    start=1
):

    new_name = (
        f"image_{number:03d}"
        f"{file.suffix}"
    )

    file.rename(
        folder / new_name
    )

The format {number:03d} produces:

image_001.jpg
image_002.jpg
image_003.jpg

21. The shutil Module

The shutil module provides high-level operations for copying, moving, and deleting files and directories.

import shutil

Important operations include:

• shutil.copy()
• shutil.copy2()
• shutil.copytree()
• shutil.move()
• shutil.rmtree()

22. Copying a File

import shutil

shutil.copy(
    "source/report.pdf",
    "backup/report.pdf"
)

The source file remains in its original location.

23. copy() vs copy2()

shutil.copy2(
    "source/report.pdf",
    "backup/report.pdf"
)

24. Moving Files

import shutil

shutil.move(
    "downloads/report.pdf",
    "documents/report.pdf"
)

Moving transfers the file to another location.

25. Copying an Entire Directory

import shutil

shutil.copytree(
    "project",
    "project_backup"
)

This recursively copies a directory and its contents.

When the destination already exists, modern Python versions support controlled use of dirs_exist_ok=True:

shutil.copytree(
    "project",
    "project_backup",
    dirs_exist_ok=True
)

26. Deleting Files

A file can be deleted using Path.unlink().

from pathlib import Path

file = Path("temp.txt")

if file.exists():
    file.unlink()

27. Deleting an Empty Directory

from pathlib import Path

folder = Path("empty_folder")

if folder.exists():
    folder.rmdir()

rmdir() only removes an empty directory.

28. Removing a Directory Tree with shutil.rmtree()

import shutil

shutil.rmtree(
    "temporary_folder"
)

29. Directory Traversal with os.walk()

os.walk() recursively traverses a directory tree.

import os

for root, directories, files in os.walk("data"):

    print("Folder:", root)

    for directory in directories:
        print("  Directory:", directory)

    for file in files:
        print("  File:", file)

This is particularly useful when processing large directory structures.

30. Find All CSV Files with os.walk()

import os

for root, directories, files in os.walk("data"):

    for file in files:

        if file.lower().endswith(".csv"):

            full_path = os.path.join(
                root,
                file
            )

            print(full_path)

31. Recursive Search with pathlib

The same task can often be expressed more concisely using rglob().

from pathlib import Path

for file in Path("data").rglob("*.csv"):
    print(file)

32. Automatic File Organization

A common real-world automation task is organizing files according to their extensions.

Example:

Downloads/
    report.pdf
    image.jpg
    data.csv
    presentation.pptx
    notes.txt

Automatically organize them into:

Downloads/
    PDFs/
    Images/
    CSV/
    Presentations/
    Text/

33. Building a File Organizer

from pathlib import Path
import shutil

source = Path("Downloads")

categories = {
    ".pdf": "PDFs",
    ".jpg": "Images",
    ".jpeg": "Images",
    ".png": "Images",
    ".csv": "CSV",
    ".xlsx": "Spreadsheets",
    ".pptx": "Presentations",
    ".docx": "Documents",
    ".txt": "Text"
}

for file in source.iterdir():

    if not file.is_file():
        continue

    extension = file.suffix.lower()

    if extension in categories:

        folder_name = categories[
            extension
        ]

        destination_folder = (
            source / folder_name
        )

        destination_folder.mkdir(
            exist_ok=True
        )

        destination = (
            destination_folder / file.name
        )

        shutil.move(
            str(file),
            str(destination)
        )

34. Safer File Organization

A production-quality organizer should consider what happens when a destination file with the same name already exists.

destination = (
    destination_folder / file.name
)

if destination.exists():

    print(
        "Skipping existing file:",
        destination
    )

else:

    shutil.move(
        str(file),
        str(destination)
    )

This avoids blindly overwriting an existing destination.

35. Creating a Reusable File Organizer

from pathlib import Path
import shutil


def organize_files(source_folder):

    source = Path(source_folder)

    categories = {
        ".pdf": "PDFs",
        ".jpg": "Images",
        ".png": "Images",
        ".csv": "CSV",
        ".xlsx": "Spreadsheets",
        ".docx": "Documents",
        ".txt": "Text"
    }

    for file in source.iterdir():

        if not file.is_file():
            continue

        category = categories.get(
            file.suffix.lower()
        )

        if category is None:
            continue

        destination_folder = (
            source / category
        )

        destination_folder.mkdir(
            exist_ok=True
        )

        destination = (
            destination_folder / file.name
        )

        if not destination.exists():

            shutil.move(
                str(file),
                str(destination)
            )


organize_files("Downloads")

36. Use a Dry Run Before Making Changes

A dry run shows what the program intends to do without actually changing files.

from pathlib import Path

source = Path("Downloads")

for file in source.iterdir():

    if file.is_file():

        print(
            "Would process:",
            file
        )

37. Checking File Size

from pathlib import Path

file = Path("data/report.pdf")

size = file.stat().st_size

print(
    "Size:",
    size,
    "bytes"
)

Convert bytes to megabytes:

size_mb = (
    file.stat().st_size
    / (1024 * 1024)
)

print(
    f"{size_mb:.2f} MB"
)

38. File Metadata

The stat() method provides file metadata.

info = file.stat()

print(info.st_size)
print(info.st_mtime)

Common metadata includes:

• File size
• Modification time
• Access time
• Creation-related metadata depending on the operating system

39. Practical Task — Find Large Files

Suppose you want to identify files larger than 100 MB.

from pathlib import Path

folder = Path("data")

limit = 100 * 1024 * 1024

for file in folder.rglob("*"):

    if not file.is_file():
        continue

    if file.stat().st_size > limit:

        print(
            "Large file:",
            file
        )

40. Finding All Files of a Specific Type

from pathlib import Path

folder = Path("documents")

pdf_files = list(
    folder.rglob("*.pdf")
)

print(
    "PDF files:",
    len(pdf_files)
)

for file in pdf_files:
    print(file)

41. Searching for Multiple File Types

from pathlib import Path

folder = Path("data")

extensions = {
    ".csv",
    ".xlsx",
    ".json"
}

for file in folder.rglob("*"):

    if (
        file.is_file()
        and file.suffix.lower()
        in extensions
    ):

        print(file)

42. Reading Environment Variables

The os module can access environment variables.

import os

username = os.getenv(
    "USERNAME"
)

print(username)

Environment variables can be useful for configuration values that should not be hard-coded into scripts.

43. Cross-Platform Home Directory

from pathlib import Path

home = Path.home()

print(home)

Path helps avoid hard-coding operating-system-specific path separators.

44. Cross-Platform File Paths

Windows, Linux, and macOS use different path conventions.

Avoid manually constructing paths such as:

folder + "\\" + filename

Prefer:

from pathlib import Path

path = Path(folder) / filename

45. Practical Automation — Create a Backup

shutil.copytree() can be used to create a directory backup.

from pathlib import Path
import shutil

source = Path("project")
backup = Path("project_backup")

shutil.copytree(
    source,
    backup,
    dirs_exist_ok=True
)

print("Backup completed.")

46. Reusable Backup Function

from pathlib import Path
import shutil


def backup_folder(
    source_folder,
    backup_folder
):

    source = Path(source_folder)
    backup = Path(backup_folder)

    if not source.exists():
        raise FileNotFoundError(
            f"Source not found: {source}"
        )

    shutil.copytree(
        source,
        backup,
        dirs_exist_ok=True
    )

    print(
        f"Backup created at: {backup}"
    )


backup_folder(
    "project",
    "project_backup"
)

47. Organizing Files by Date

File-system automation can combine file metadata with directory creation.

For example, files could be organized into:

Archive/
    2026/
        08/
        09/
        10/

The exact date-based logic depends on the metadata and business requirement.

48. Add Logging to Automation Scripts

Production automation should record what it does.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

logging.info(
    "File organization started."
)

logging.info(
    "Processing completed."
)

Logging is more useful than relying only on print() statements for larger automation systems.

49. Error Handling in File Automation

File operations can fail because of missing files, permissions, invalid paths, locked files, or other operating-system conditions.

from pathlib import Path

file = Path("data/report.txt")

try:

    content = file.read_text()

except FileNotFoundError:

    print("File does not exist.")

except PermissionError:

    print("Permission denied.")

except OSError as error:

    print(
        "Operating system error:",
        error
    )

50. Safe File Deletion Pattern

from pathlib import Path

file = Path("temporary.txt")

if (
    file.exists()
    and file.is_file()
):

    confirm = input(
        f"Delete {file}? (yes/no): "
    )

    if confirm.lower() == "yes":
        file.unlink()
        print("Deleted.")

Interactive confirmation can be appropriate for manual utilities, while scheduled automation should generally use carefully defined rules and logging rather than an interactive prompt.

51. Finding Temporary Files

A script can identify files using naming conventions.

from pathlib import Path

folder = Path("project")

for file in folder.rglob("*"):

    if (
        file.is_file()
        and file.name.endswith(".tmp")
    ):

        print(
            "Temporary file:",
            file
        )

After identifying them, decide carefully whether they should be archived, moved, or deleted.

52. Building a Directory Tree Viewer

from pathlib import Path


def show_tree(
    folder,
    level=0
):

    folder = Path(folder)

    for item in sorted(
        folder.iterdir()
    ):

        print(
            "    " * level
            + item.name
        )

        if item.is_dir():

            show_tree(
                item,
                level + 1
            )


show_tree("project")

This demonstrates recursive directory traversal.

53. Understanding Recursive Directory Traversal

A directory may contain directories, which may contain more directories.

Project
├── data
│   ├── raw
│   └── processed
├── reports
│   ├── monthly
│   └── annual
└── scripts

A recursive traversal visits each directory and then processes its children.

54. os vs pathlib vs shutil

55. Real-World Automation Examples

56. File Automation in Data Science

File-system automation is especially important in Data Science pipelines.

Consider a project containing:

project/
├── data/
│   ├── raw/
│   ├── processed/
│   └── external/
├── notebooks/
├── reports/
├── outputs/
└── scripts/

Python can automate movement of data between these stages.

57. Mini Project — Automatic Download Organizer

Build a utility that scans a folder and organizes files into category folders.

from pathlib import Path
import shutil


SOURCE = Path("Downloads")

CATEGORIES = {
    ".pdf": "PDFs",
    ".jpg": "Images",
    ".jpeg": "Images",
    ".png": "Images",
    ".gif": "Images",
    ".csv": "Data",
    ".xlsx": "Spreadsheets",
    ".docx": "Documents",
    ".pptx": "Presentations",
    ".txt": "Text"
}


def organize_downloads():

    if not SOURCE.exists():
        print("Downloads folder not found.")
        return

    for file in SOURCE.iterdir():

        if not file.is_file():
            continue

        extension = (
            file.suffix.lower()
        )

        category = CATEGORIES.get(
            extension
        )

        if category is None:
            continue

        destination_folder = (
            SOURCE / category
        )

        destination_folder.mkdir(
            exist_ok=True
        )

        destination = (
            destination_folder / file.name
        )

        if destination.exists():

            print(
                "Skipped:",
                file.name
            )

            continue

        shutil.move(
            str(file),
            str(destination)
        )

        print(
            f"Moved: {file.name} "
            f"→ {category}"
        )


organize_downloads()

58. Mini Project — Enhancement Challenges

Improve the Download Organizer by adding:

1. Dry-run mode.
2. Logging.
3. Duplicate-name handling.
4. Recursive folder scanning.
5. File-size reporting.
6. Summary of moved files.
7. Error handling.
8. Custom category configuration.
9. Command-line arguments.
10. Scheduled execution.

59. File Automation Interview Questions

from pathlib import Path

path = Path("report.pdf")

print(path.exists())

Path(
    "data/2026/reports"
).mkdir(
    parents=True,
    exist_ok=True
)

60. Examination Questions — MCQs

61. Practical Examination Questions

from pathlib import Path

folder = Path("data")

for item in folder.iterdir():

    if item.is_file():
        print(item)

from pathlib import Path

for file in Path("data").rglob("*.csv"):
    print(file)

from pathlib import Path

Path(
    "data/raw"
).mkdir(
    parents=True,
    exist_ok=True
)

Path(
    "data/processed"
).mkdir(
    parents=True,
    exist_ok=True
)

from pathlib import Path

folder = Path("documents")

for file in folder.glob("*.txt"):

    new_name = (
        f"{file.stem}_processed"
        f"{file.suffix}"
    )

    file.rename(
        file.with_name(new_name)
    )

import shutil

shutil.copytree(
    "project",
    "project_backup",
    dirs_exist_ok=True
)

62. Important Tips for File-System Automation

• Prefer pathlib for clear path manipulation in new code.
• Use shutil for high-level copying and moving.
• Use os.walk() when you need explicit recursive directory traversal.
• Use glob() and rglob() for pattern-based file discovery.
• Always check whether a path exists before performing sensitive operations.
• Do not blindly overwrite files.
• Test bulk operations on sample data.
• Use a dry-run mode for potentially destructive scripts.
• Add logging to long-running or production automation.
• Handle expected exceptions such as FileNotFoundError and PermissionError.
• Avoid hard-coded operating-system-specific path separators.
• Keep source data backed up before large-scale transformations.

63. Quick-Fire Interview Revision

64. Final Takeaways

• Python can automate repetitive file and directory operations.
• The os module provides operating-system interaction capabilities.
• pathlib provides an expressive object-oriented interface for filesystem paths.
• shutil provides high-level copying, moving, and directory operations.
• glob() and rglob() are useful for pattern-based file discovery.
• os.walk() is useful for recursive directory traversal.
• Bulk renaming can eliminate repetitive manual work.
• File organizers can classify files by extension or other business rules.
• Backups can be automated with shutil.copytree().
• Destructive operations should use verification, testing, logging, and appropriate safeguards.
• Cross-platform scripts should avoid hard-coded path separators.
• File-system automation is an important foundation for larger Data Science and ETL pipelines.