Sys, subprocess, and argparse

---

🔧 Python’s sys Module

The sys module lets you interact with the Python runtime environment (the interpreter, I/O streams, CLI args, exit codes, import paths, etc.).

---

Command-Line Arguments: sys.argv

sys.argv holds the arguments passed to your script from the command line.

Example script: script.py

import sys

print("Arguments:", sys.argv)

Run it:

python script.py hello 123 world

Output:

Arguments: ['script.py', 'hello', '123', 'world']

Use case: reading user input to the script

import sys

if len(sys.argv) < 3:
    print("Usage: python downloader.py <url> <output>")
    sys.exit(1)

url = sys.argv[1]
output = sys.argv[2]

print(f"Downloading {url} to {output}")

---

Exiting the Program: sys.exit()

You can exit with a success code (0) or an error code (non-zero).

import sys

if not sys.platform.startswith("linux"):
    print("❌ This script only runs on Linux.")
    sys.exit(1)   # exit with error code

print("Running...")

Why use this?

• End execution early

• Communicate success/failure to the OS

• CI/CD pipelines depend on exit codes

---

Stdout / Stderr / Stdin

These represent the shell’s:

• stdout → normal output

• stderr → error output

• stdin → input stream

Example: writing to stdout

import sys

sys.stdout.write("This is normal output\n")

Example: writing to stderr

import sys

sys.stderr.write("This is an error message\n")

This matters because many tools capture only stdout while showing stderr separately.

Example: reading from stdin

import sys

data = sys.stdin.read()
print("You entered:", data)

Run it:

echo "hello world" | python script.py

Output:

You entered: hello world

---

System Information

sys.version, sys.platform, and others let you inspect the runtime environment.

Example: checking Python version

import sys

if sys.version_info < (3, 10):
    print("You need Python 3.10 or higher!")
    sys.exit(1)

Example: get OS platform

import sys

print(sys.platform)

Typical outputs:

• "darwin" → macOS

• "linux" → Linux

• "win32" → Windows

Example: find the Python interpreter path

import sys

print(sys.executable)

This helps when debugging virtual environments.

---

Manipulating sys.path (Import Search Path)

sys.path is the list of directories Python searches when importing modules.

Example: print module search paths

import sys

for p in sys.path:
    print(p)

Example: add a custom path

import sys

sys.path.append("/opt/shared-libs")

import my_shared_library

Useful when:

• working with monorepos

• local development of reusable modules

• running scripts without packaging them

---

sys.getsizeof() — Memory Size of Objects

This shows memory usage of Python objects.

Example:

import sys

print(sys.getsizeof(100))            # small int
print(sys.getsizeof("hello world")) # string size
print(sys.getsizeof([1,2,3,4,5]))   # list size

This is very important for Data Engineering when:

• optimizing memory

• checking object sizes in pipelines

• diagnosing memory leaks

---

sys.modules — Loaded Module Cache

Shows all imported modules and their references.

Example:

import sys
import math

print("Is math loaded?", "math" in sys.modules)

Useful when debugging imports or circular imports.

---

🏃 subprocess Module — Run External Commands

subprocess is for running external programs from Python. This is how you run shell commands like ls, grep, ping, ps, etc.

It replaced older modules like os.system, commands, popen2.

⭐ The Two Most Important Functions

1. subprocess.run()

The simplest and safest API.

import subprocess

result = subprocess.run(
    ["ls", "-l"],
    capture_output=True,
    text=True
)

print(result.stdout)

✔ Runs command ✔ Waits for it to finish ✔ Captures output ✔ Returns a CompletedProcess

---

2. subprocess.Popen()

More advanced, for streaming output or interactive programs.

from subprocess import Popen, PIPE

p = Popen(["ping", "-c", "3", "google.com"], stdout=PIPE, text=True)

for line in p.stdout:
    print(line, end="")

Use Popen when you need:

• long-running processes

• stream output live

• send input to process (stdin)

• more control over pipes

---

🧪 Examples You’ll Actually Use

✔ Example: Run a Shell Command

subprocess.run("echo Hello", shell=True)

⚠️ Only use shell=True if necessary (security risk with user input).

---

✔ Example: Check if Command Succeeded

result = subprocess.run(["python", "--version"])
if result.returncode == 0:
    print("Success!")

---

✔ Example: Capture stdout + stderr

result = subprocess.run(
    ["ls", "/not/here"],
    capture_output=True,
    text=True
)

print(result.stderr)

---

✔ Example: Send Input to a Process

subprocess.run(
    ["grep", "hello"],
    input="hello world\nhi",
    text=True
)

---

Argparse

argparse is the standard Python library for building command-line interfaces (CLIs). Whenever you run a command like:

python my_script.py --input file.csv --verbose

you’re using the kind of functionality argparse helps you build.

It converts command-line arguments → Python variables, supports validation, help messages, types, defaults, and much more.

Let’s break it down clearly and practically.

---

🔧 What argparse Does

✔ Defines which arguments your script accepts ✔ Parses them from the command line ✔ Automatically generates --help text ✔ Validates types and allowed values ✔ Supports flags (--verbose), options (--output file), and positional arguments

---

⭐ Minimal Example

script.py

import argparse

parser = argparse.ArgumentParser()

parser.add_argument("name")  # positional argument

args = parser.parse_args()

print("Hello,", args.name)

Run:

python script.py Nariman

Output:

Hello, Nariman

---

⭐ Example With Flags and Options

import argparse

parser = argparse.ArgumentParser(description="A demo CLI")

parser.add_argument("--count", type=int, default=1)
parser.add_argument("--verbose", action="store_true")

args = parser.parse_args()

for _ in range(args.count):
    print("Hello")

if args.verbose:
    print("Done printing!")

Run:

python script.py --count 3 --verbose

Output:

Hello
Hello
Hello
Done printing!

---

🔥 What You Can Add Using argparse

Positional Arguments

Required and order-based.

parser.add_argument("input_file")

Use:

python app.py data.csv

---

Optional Arguments (“Flags”)

Boolean flag:

parser.add_argument("--debug", action="store_true")

Run:

python app.py --debug

Option with a value:

parser.add_argument("--threads", type=int)

Run:

python app.py --threads 8

---

Argument Types

parser.add_argument("--ratio", type=float)
parser.add_argument("--date", type=lambda s: datetime.fromisoformat(s))

---

Choices (Restrict allowed values)

parser.add_argument("--mode", choices=["fast", "slow", "turbo"])

Run:

python app.py --mode slow

---

Default Values

parser.add_argument("--retries", type=int, default=3)

---

Lists / Multiple Values

Method 1 — nargs

parser.add_argument("--items", nargs="+")

Run:

python app.py --items a b c

Output:

args.items  # ['a', 'b', 'c']

Method 2 — repeat flags

parser.add_argument("--tag", action="append")

Run:

python app.py --tag v1 --tag v2

Output:

['v1', 'v2']

---

Required Flags

parser.add_argument("--url", required=True)

---

Help and Description

parser = argparse.ArgumentParser(
    description="Download files from the internet."
)

parser.add_argument("--url", help="URL of the file")

The user gets automatic documentation:

python app.py --help

Output:

usage: app.py [-h] --url URL

Download files from the internet.

options:
  -h, --help     show this help message
  --url URL      URL of the file

---

🧱 Complex Example (Realistic)

import argparse

parser = argparse.ArgumentParser(description="ETL Pipeline")

parser.add_argument("input")
parser.add_argument("output")

parser.add_argument("--format", choices=["csv", "json", "parquet"], default="csv")
parser.add_argument("--limit", type=int)
parser.add_argument("--verbose", action="store_true")

args = parser.parse_args()

print("Reading:", args.input)
print("Writing:", args.output)
print("Format:", args.format)

Run:

python etl.py input.csv out.parquet --format parquet --verbose

---