Virtual Environment and Python package manager

Virtual Environment and Python package manager

Keeping Each Project's Code and Tools Separate for Easy Management

A virtual environment in Python is a self-contained directory that allows you to isolate a project's dependencies from the global Python environment. This ensures that each project can use its versions of libraries without affecting other projects, solving issues like dependency conflicts or version mismatches.

The diagram shows a Python Virtual Environment concept using three different systems:

  1. System 1: Inventory Management

  2. System 2: Accounting System

  3. System 3: HR System

Each system uses its own Virtual Environment. Let me explain how this works in simple terms:

  • System 1 (Inventory Management) creates its own virtual environment. This is like creating a small isolated space where all the packages and tools needed for the inventory system are installed.

  • System 2 (Accounting System) also has its own virtual environment. This environment will contain only the tools required for the accounting system, preventing any interference with other systems.

  • System 3 (HR System) has a separate virtual environment, where only the HR system's tools and libraries exist.

virtual environments help to keep things organized by giving each project its own separate workspace, preventing problems caused by mixing different tools or versions between projects​(Python Land)​(Deepgram)​(Software Engineering Blog)


Why virtual environments are important:

  1. Dependency Management: Each project can have its own set of dependencies, ensuring that the right versions of libraries are used for that specific project.

  2. Isolation: By isolating projects, you avoid conflicts between packages, especially if you’re working on multiple projects that require different versions of the same library.

  3. Cleaner Development Workflow: Virtual environments help keep your global Python installation clean by only installing project-specific dependencies in a controlled environment.

How to Use Virtual Environments:

  • Create a Virtual Environment: You can use the built-in venv module to create a virtual environment. Run the following command in your terminal:

      python -m venv myenv
    

    This creates a folder (myenv) that contains a Python interpreter and a separate package directory.

  • Activate the Virtual Environment:

    • On Windows: .\myenv\Scripts\activate

    • On Mac/Linux: source myenv/bin/activate After activation, the command prompt will show the name of your virtual environment, indicating that the environment is active.

  • Install Dependencies: Once the environment is active, any package you install using pip (pip install package_name) will only be available within that environment.

  • Deactivate the Environment: After you finish working, you can deactivate the virtual environment by simply typing deactivate.

Managing Virtual Environments with Tools like Pipenv and Poetry:

  • Pipenv: Combines virtual environment management with package management, making it easier to handle dependencies in collaborative projects.

  • Poetry: Offers more advanced dependency management and packaging capabilities while also managing virtual environments, making it ideal for projects that require complex packaging​(Python Land)​(Deepgram)​(DEV Community).

Incorporating virtual environments into your workflow can greatly simplify project management and help maintain a stable, conflict-free development environment.


Python package manager

A Python package manager is a tool that helps developers install, manage, and update external libraries and dependencies for their projects. The most widely known package manager is pip (which stands for "Pip Installs Packages"). Pip is the default and standard tool for managing packages from the Python Package Index (PyPI), which is a repository containing thousands of Python libraries. Pip simplifies the installation of packages and handles dependencies by resolving and downloading any needed packages automatically​(Dhruv Awasthi)​(Inedo Blog).

While pip is essential and widely used, there are other popular package managers that provide additional features:

  1. Pipenv: This combines package management and virtual environment management. It is preferred for collaborative projects because it simplifies managing dependencies and creating isolated environments. It’s great for projects needing both development and production environments​(DEV Community)​(Inedo Blog).

  2. Poetry: A more modern package manager that not only handles dependencies but also helps build and distribute Python applications. It uses a pyproject.toml file for configuration and is popular because it simplifies packaging and dependency management for both small and large projects​(Jumping Rivers)​(Slant).

  3. Conda: Unlike pip, Conda is a more versatile package manager designed not only for Python but also for other languages. It is popular in data science environments and is known for managing libraries and packages for scientific computing. Conda can also manage virtual environments similar to pip and Pipenv​(Inedo Blog).

Each of these package managers has its strengths. For example, Pipenv and Poetry focus more on ensuring smooth workflows for development environments, while Conda is highly favored in scientific computing environments due to its ability to manage binary packages more effectively​(Jumping Rivers)​(Slant).

For any Python project, choosing the right package manager depends on your specific needs, such as collaboration, environment isolation, or performance requirements.


Ensure you don’t upload your virtual environment (venv)

to GitHub or any cloud

How to manage your dependencies correctly:

Steps:

  1. Use .gitignore to ignore the virtual environment: Create or modify a .gitignore file in the root of your project and add the following line to ensure your virtual environment folder is not pushed to GitHub:

     # Ignore the virtual environment folder
     venv/
    

    This will prevent Git from tracking your venv folder, ensuring it’s not included in your commits or pushed to your repository.

  2. Generate a requirements.txt file: When you're ready to push your code, save all the current dependencies (packages you’ve installed in the virtual environment) in a requirements.txt file. This allows anyone who clones your repository to install the necessary dependencies.

    Run this command:

     pip freeze > requirements.txt
    
    • This will create a requirements.txt file with all the packages and their versions.
  3. Clone or update the repository: When someone (or you) clones or pulls updates from the repository, instead of including the venv, they can recreate the same environment using the requirements.txt file.

    After cloning or pulling the repository, run:

     pip install -r requirements.txt
    

    This installs all the dependencies listed in the requirements.txt file in your new virtual environment.

AGAIN

  • Use .gitignore to exclude the venv folder.

  • Use pip freeze > requirements.txt to save all the packages you’ve installed.

  • Use pip install -r requirements.txt to install dependencies when cloning or updating the project.

This is a best practice to avoid burdening your repository with unnecessary files while ensuring a smooth setup for others who work on your project.