python-telegram-bot Library

December 12, 2024 ~ Davis Miller ~ Leave a comment

python-telegram-bot is a popular and robust Python library for building Telegram bots using the Telegram Bot API. It simplifies many aspects of communicating with the API, handling updates, parsing messages, and implementing bot logic. With python-telegram-bot, developers can focus on their bot's functionality rather than dealing with low-level HTTP requests and JSON parsing. The library is open-source and widely used, with a large community and comprehensive documentation.

Key Features

Full Wrapper Around Telegram Bot API:
python-telegram-bot covers nearly all features of the Telegram Bot API, enabling you to send and receive messages, media, manage groups and channels, create inline keyboards, and more.
Extensive Documentation & Support:
The library is well-documented, with a detailed wiki, numerous examples, and active community support via GitHub issues and a Telegram support group.
Async and Sync Support:
With the release of v13 and beyond, python-telegram-bot supports both traditional synchronous operations and asyncio-based asynchronous code (introduced in v20), allowing for scalable, high-performance bots.
Update Handling with Different Models:
- Polling: Convenient for development and smaller bots. The bot sends getUpdates requests to Telegram and processes incoming updates.
- Webhooks: For production or performance-sensitive setups, you can set up a webhook so Telegram pushes updates to your server in real-time. The library can run its own webserver or integrate with frameworks like Flask or Django.
Command and Message Handlers:
python-telegram-bot provides a Dispatcher and a rich set of handlers and filters (e.g., CommandHandler, MessageHandler, CallbackQueryHandler) that map specific message patterns, commands, or callback data to your handling functions.
Inline Queries and Keyboards:
Inline queries and inline keyboards are well-supported. The library provides classes and methods to create InlineKeyboardButtons, InlineKeyboardMarkup, and handle callbacks easily.
ConversationHandler:
A powerful feature to manage multi-step conversations. You can define states and transitions, making it straightforward to build guided user flows, forms, or interactive dialogs.
Persistent Storage:
Supports storing bot data, chat data, and user data across sessions using built-in persistence classes for different backends (like PicklePersistence) or custom persistence methods.

Installation

You can install python-telegram-bot using pip:

pip install python-telegram-bot

For async version (from v20 onwards), no special installation is needed since async support is included by default.

Basic Usage Example (Synchronous)

Here's a simple bot that responds to the /start command with a greeting:

import logging
from telegram import Update
from telegram.ext import Updater, CommandHandler, CallbackContext

# Enable logging
logging.basicConfig(format='%(asctime)s – %(name)s – %(levelname)s – %(message)s', level=logging.INFO)

def start_command(update: Update, context: CallbackContext):
update.message.reply_text("Hello! I am your bot. How can I help you today?")

def main():
# Replace 'YOUR_API_TOKEN' with your bot's token from BotFather
updater = Updater("YOUR_API_TOKEN", use_context=True)
dispatcher = updater.dispatcher

# Add a CommandHandler for /start
dispatcher.add_handler(CommandHandler("start", start_command))

# Start polling for updates
updater.start_polling()
# Run until you press Ctrl-C
updater.idle()

if __name__ == '__main__':
main()

How it works:

Updater: Manages the connection to Telegram via long polling or webhooks, and channels updates to the Dispatcher.
Dispatcher: Distributes incoming updates to handlers based on filters or commands.
CommandHandler("start", start_command): Calls start_command whenever a user sends /start.

Using the Async Version (from v20 onwards)

Starting with v20, python-telegram-bot leverages asyncio. A similar bot using async calls might look like this:

import asyncio
import logging
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes

logging.basicConfig(level=logging.INFO)

async def start_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text("Hello! I am your async bot.")

async def main():
app = ApplicationBuilder().token("YOUR_API_TOKEN").build()

app.add_handler(CommandHandler("start", start_command))

await app.run_polling()

if __name__ == '__main__':
asyncio.run(main())

Here, ApplicationBuilder creates the bot application, and run_polling() is an async method that continuously fetches updates. Handlers and callbacks are async, and you use await when sending messages or performing other I/O tasks.

Handlers and Filters

A key strength of python-telegram-bot is the variety of handlers and filters:

CommandHandler: Triggers on /command messages.
MessageHandler: Matches text messages, media, or other content using filters.
CallbackQueryHandler: Handles button presses on inline keyboards.
InlineQueryHandler: Handles inline queries when a user types @YourBot in any chat.

Filters can limit which messages a handler should process. For example, Filters.text & ~Filters.command matches any text message that's not a command:

from telegram.ext import MessageHandler, filters

dispatcher.add_handler(MessageHandler(filters.Text() & ~filters.COMMAND, text_handler))

Inline Keyboards

To send a message with an inline keyboard:

from telegram import InlineKeyboardButton, InlineKeyboardMarkup

async def ask_question(update: Update, context: ContextTypes.DEFAULT_TYPE):
keyboard = [
[InlineKeyboardButton("Option 1", callback_data='1'),
InlineKeyboardButton("Option 2", callback_data='2')]
]
reply_markup = InlineKeyboardMarkup(keyboard)
await update.message.reply_text("Choose an option:", reply_markup=reply_markup)

async def button_callback(update: Update, context: ContextTypes.DEFAULT_TYPE):
query = update.callback_query
await query.answer() # Acknowledge the callback
choice = query.data
await query.edit_message_text(text=f"You chose option {choice}")

You would add these handlers with:

app.add_handler(CommandHandler("ask", ask_question))
app.add_handler(CallbackQueryHandler(button_callback))

ConversationHandler

For multi-step interactions, define states and transitions:

from telegram.ext import ConversationHandler, MessageHandler, CommandHandler, filters

ASKING_NAME, ASKING_AGE = range(2)

async def start_conversation(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text("What is your name?")
return ASKING_NAME

async def name_handler(update: Update, context: ContextTypes.DEFAULT_TYPE):
context.user_data['name'] = update.message.text
await update.message.reply_text("What is your age?")
return ASKING_AGE

async def age_handler(update: Update, context: ContextTypes.DEFAULT_TYPE):
age = update.message.text
name = context.user_data['name']
await update.message.reply_text(f"Nice to meet you {name}, age {age}.")
return ConversationHandler.END

async def cancel(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text("Conversation cancelled.")
return ConversationHandler.END

conv_handler = ConversationHandler(
entry_points=[CommandHandler('start', start_conversation)],
states={
ASKING_NAME: [MessageHandler(filters.TEXT & ~filters.COMMAND, name_handler)],
ASKING_AGE: [MessageHandler(filters.TEXT & ~filters.COMMAND, age_handler)],
},
fallbacks=[CommandHandler('cancel', cancel)]
)

app.add_handler(conv_handler)

This sets up a conversation flow where the user's responses guide them through different states until it ends.

Persistence

To keep data across restarts:

from telegram.ext import PicklePersistence

persistence = PicklePersistence(filepath='bot_data.pkl')
app = ApplicationBuilder().token("TOKEN").persistence(persistence).build()

Now, context.bot_data, context.chat_data, and context.user_data will be saved and restored automatically.

Webhook Setup

Instead of polling, you can set a webhook:

app = ApplicationBuilder().token("YOUR_API_TOKEN").build()
await app.bot.set_webhook("https://yourdomain.com/webhook")

app.run_webhook(
listen="0.0.0.0",
port=8443,
url_path="/webhook",
webhook_url="https://yourdomain.com/webhook",
# ssl context if needed
)

Your server will receive updates instantly. Ensure the endpoint is HTTPS and accessible by Telegram.

Error Handling and Logging

Integrate logging and error handlers:

async def error_handler(update: object, context: ContextTypes.DEFAULT_TYPE):
# Log the error
logging.error(msg="Exception while handling an update:", exc_info=context.error)

app.add_error_handler(error_handler)

This ensures you catch and log unexpected exceptions gracefully.

Common Patterns and Tips

Environment Variables: Store your bot token in an environment variable and load it at runtime for security.
Modular Code: Break your bot logic into separate modules or classes for better maintainability.
Testing Locally: Start with polling in a local environment. For production, move to webhooks.
Version Compatibility: Check the documentation for version compatibility, especially when migrating from synchronous to async versions (e.g., from v13 to v20).

Python MySQL Interaction

December 11, 2024 ~ Davis Miller ~ Leave a comment

MySQL is a popular open-source relational database management system (RDBMS), widely used for web applications, data warehousing, and more. Python, due to its simplicity and rich ecosystem, is often used to interact with MySQL databases to perform common database tasks: fetching data, inserting new records, updating rows, and running complex queries.

There are several libraries and modules that enable Python-MySQL interaction. The two most common ones are:

MySQL Connector/Python (official MySQL driver provided by Oracle)
PyMySQL (a pure-Python MySQL client library)

For this guide, we will primarily focus on MySQL Connector/Python, as it's officially supported by Oracle, the maintainers of MySQL, and doesn't require additional dependencies.

Installation

Before writing code, ensure that MySQL and the appropriate Python driver are installed.

MySQL Server:
Install MySQL server on your system. For instructions, see the MySQL official documentation.

Python Environment:
Make sure you have Python 3.x installed. You can verify by running:

python –version

MySQL Connector/Python:
Install via pip:

pip install mysql-connector-python

This command downloads and installs the MySQL Connector/Python library, allowing your Python scripts to interface with MySQL.

Connecting to the Database

To interact with a MySQL database, you need a connection object. This object represents the session between your Python code and the MySQL server.

Key Parameters Needed:

host: The MySQL server hostname or IP address. Often localhost if running on the same machine.
user: The username to authenticate with.
password: The user's password.
database: The name of the database you want to work with (optional at connection time, can also be selected later).

Example:

import mysql.connector

# Establish connection
connection = mysql.connector.connect(
host="localhost",
user="myuser",
password="mypassword",
database="mydatabase"
)

# Check if the connection was successful
if connection.is_connected():
print("Connected to MySQL database!")

Explanation:

mysql.connector.connect(…) returns a connection object if successful.
The is_connected() method checks if the connection is active.

Error Handling: If the connection fails, a mysql.connector.Error exception is raised. It's best practice to wrap the connection in a try-except block:

import mysql.connector
from mysql.connector import Error

try:
connection = mysql.connector.connect(
host="localhost",
user="myuser",
password="mypassword",
database="mydatabase"
)
if connection.is_connected():
print("Connected successfully.")
except Error as e:
print(f"Error connecting to MySQL: {e}")

The Cursor Object

Once connected, you interact with the database via a cursor object. A cursor is like a handle or pointer that you use to execute SQL commands and fetch results.

Creating a Cursor:

cursor = connection.cursor()

Explanation:

connection.cursor() returns a cursor object linked to that connection.
With this cursor, you can call execute() to run SQL statements, and fetchone() or fetchall() to retrieve query results.

Executing SQL Queries

You can execute various types of queries: SELECT (retrieving data), INSERT (adding rows), UPDATE (modifying existing rows), DELETE (removing rows), and Data Definition Language (DDL) commands like CREATE TABLE or DROP TABLE.

Example (Creating a Table):

create_table_query = """
CREATE TABLE IF NOT EXISTS employees (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100) NOT NULL,
role VARCHAR(50),
salary DECIMAL(10,2)
)
"""
cursor.execute(create_table_query)

Explanation:

We define a multi-line string with the SQL DDL command to create an employees table if it doesn't already exist.
cursor.execute() runs this SQL command. If successful, the table will be created.

Inserting Data

Example (Inserting Rows):

insert_query = "INSERT INTO employees (name, role, salary) VALUES (%s, %s, %s)"
values = ("Alice", "Engineer", 75000.00)
cursor.execute(insert_query, values)

# To persist changes to the database, commit the transaction
connection.commit()
print("Inserted 1 row into employees table.")

Explanation:

We use placeholders %s in the query and pass a tuple (name, role, salary) as values.
The driver automatically sanitizes and escapes these values, preventing SQL injection.
After execute() for INSERT/UPDATE/DELETE queries, we must commit() to save changes permanently.

Multiple Inserts at Once:

insert_query = "INSERT INTO employees (name, role, salary) VALUES (%s, %s, %s)"
multiple_values = [
("Bob", "Manager", 90000.00),
("Charlie", "Engineer", 70000.00),
("Diana", "HR Specialist", 65000.00)
]
cursor.executemany(insert_query, multiple_values)
connection.commit()
print(f"Inserted {cursor.rowcount} rows into employees table.")

Explanation:

executemany() executes the given query for each tuple in the list, inserting multiple rows in a single batch operation.
cursor.rowcount tells how many rows were affected by the last operation.

Selecting Data (Fetching Rows)

Example (Selecting Rows):

select_query = "SELECT id, name, role, salary FROM employees"
cursor.execute(select_query)

# Fetch all rows returned by the query
rows = cursor.fetchall()

for row in rows:
print(row)

Explanation:

fetchall() returns a list of tuples, where each tuple represents a row from the result set.

Example output might be:

(1, 'Alice', 'Engineer', Decimal('75000.00'))
(2, 'Bob', 'Manager', Decimal('90000.00'))
(3, 'Charlie', 'Engineer', Decimal('70000.00'))
(4, 'Diana', 'HR Specialist', Decimal('65000.00'))

Other Fetch Methods:

fetchone(): retrieves the next row from the result, or None if no more rows are available.
fetchmany(size): retrieves the next size rows from the result.

Iterating with fetchone():

cursor.execute("SELECT name, role FROM employees")
row = cursor.fetchone()
while row is not None:
print(row)
row = cursor.fetchone()

Updating Data

Example (Updating Rows):

update_query = "UPDATE employees SET salary = %s WHERE name = %s"
values = (80000.00, "Charlie")
cursor.execute(update_query, values)
connection.commit()
print(f"Updated {cursor.rowcount} row(s).")

Explanation:

We update the salary of the employee named "Charlie" to 80000.00.
Always commit() after INSERT, UPDATE, or DELETE to make changes persistent.

Deleting Data

Example (Deleting Rows):

delete_query = "DELETE FROM employees WHERE name = %s"
value = ("Diana",)
cursor.execute(delete_query, value)
connection.commit()
print(f"Deleted {cursor.rowcount} row(s).")

Explanation:

%s placeholders are used for parameter substitution.
We commit the transaction to finalize the deletion.

Preventing SQL Injection

Parameter Binding:

Always use parameterized queries with %s placeholders and separate values tuples.
Never build SQL queries by string concatenation, e.g., f"SELECT * FROM employees WHERE name = '{user_input}'".
Using execute() with parameters ensures that the driver escapes input to protect against SQL injection.

Example (Secure Query):

user_input = "Bob'; DROP TABLE employees;–" # a malicious attempt
query = "SELECT * FROM employees WHERE name = %s"
cursor.execute(query, (user_input,))

Because we used parameterized queries, the malicious part is treated as a literal string, not executable SQL.

Transactions and Commits

MySQL, by default, commits changes after each statement if autocommit is True. With MySQL Connector/Python, autocommit is off by default, meaning you need to explicitly call connection.commit().

Example:

connection.start_transaction()
cursor.execute("UPDATE employees SET salary = 100000 WHERE name = 'Bob'")
cursor.execute("UPDATE employees SET salary = 70000 WHERE name = 'Charlie'")
connection.commit() # Both updates are committed together

If something goes wrong:

connection.rollback() # Revert all changes since last commit

Explanation:

start_transaction() explicitly begins a transaction.
commit() finalizes all operations since the start of the transaction.
rollback() reverses them if an error occurs.

Handling Errors and Exceptions

When things go wrong (e.g., invalid queries, lost connections, permission issues), mysql.connector.Error exceptions are raised.

Example:

from mysql.connector import Error

try:
cursor.execute("SELECT * FROM non_existent_table")
rows = cursor.fetchall()
except Error as e:
print(f"An error occurred: {e}")

Explanation:

Always catch Error exceptions to handle unexpected failures gracefully.
This could mean logging the error, alerting a user, or retrying the operation.

Connection Pooling

For highly concurrent applications (e.g., web servers), creating and closing connections frequently is inefficient. Connection pooling reuses established connections to improve performance.

Using MySQL Connector/Python's Pooling:

from mysql.connector import pooling

pool = pooling.MySQLConnectionPool(
pool_name="mypool",
pool_size=5,
host="localhost",
user="myuser",
password="mypassword",
database="mydatabase"
)

# Get a connection from the pool
connection = pool.get_connection()
cursor = connection.cursor()
cursor.execute("SELECT * FROM employees")
…

Explanation:

MySQLConnectionPool creates a pool of connections that can be reused.
Instead of creating a new connection each time, get_connection() fetches one from the pool.
Improves performance for applications that handle multiple parallel requests.

Working with Different Data Types

Date/Time Types:

MySQL date/time columns (DATE, DATETIME, TIMESTAMP) can be fetched as Python datetime.date and datetime.datetime objects.
Inserting Python datetime objects is also straightforward via parameter substitution.

Example:

import datetime

insert_query = "INSERT INTO employees (name, role, salary, hire_date) VALUES (%s, %s, %s, %s)"
values = ("Eve", "Intern", 40000.00, datetime.datetime(2024, 1, 1))
cursor.execute(insert_query, values)
connection.commit()

JSON Fields (MySQL 5.7+):

MySQL supports a JSON column type. The connector returns JSON data as strings (by default). You can parse it with json.loads() in Python.

Using Stored Procedures

You can invoke stored procedures defined in MySQL. Stored procedures encapsulate complex business logic within the database.

Example (Calling a Stored Procedure):

# Suppose we have a stored procedure: CREATE PROCEDURE get_employees() SELECT * FROM employees;
cursor.callproc('get_employees')

# callproc returns a list of cursors
for result in cursor.stored_results():
rows = result.fetchall()
for row in rows:
print(row)

Explanation:

callproc() runs the named stored procedure.
cursor.stored_results() yields result sets if the procedure returns any.

Performance Considerations

Indexing: Ensure your MySQL tables have appropriate indexes for fast lookups.
Batch Operations: Use executemany() for bulk inserts to reduce round-trip times.
Connection Management: Avoid opening and closing connections repeatedly; use a persistent connection or connection pooling.
Fetch Size: For very large result sets, consider fetchmany() or streaming results to manage memory usage.

Security Best Practices

Use Least Privileged Accounts: Connect to MySQL with a user that has the minimum required permissions (no unnecessary GRANT privileges).
SSL/TLS: For production environments, use SSL/TLS connections to encrypt data in transit.
Rotation of Credentials: Change database passwords regularly.
Secure Storage of Credentials: Do not hardcode credentials in your Python code. Use environment variables, configuration files secured with appropriate permissions, or Azure Key Vault/AWS Secrets Manager if on the cloud.

Example Application Flow

Below is a hypothetical scenario that ties all these concepts together:

Scenario: A Python script that manages an Employee database. It connects to MySQL, inserts data from a CSV file, updates salaries, and retrieves reports.

Pseudo-code:

import csv
import mysql.connector
from mysql.connector import Error

def load_employees_from_csv(filename):
employees = []
with open(filename, newline=") as f:
reader = csv.reader(f)
# Assuming CSV has name,role,salary columns
for row in reader:
name, role, salary_str = row
employees.append((name, role, float(salary_str)))
return employees

try:
connection = mysql.connector.connect(
host="localhost",
user="myuser",
password="mypassword",
database="mydatabase"
)
cursor = connection.cursor()

# Create table if not exists
cursor.execute("""
CREATE TABLE IF NOT EXISTS employees (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100),
role VARCHAR(50),
salary DECIMAL(10,2)
)
""")

# Insert employees from CSV
new_employees = load_employees_from_csv("employees.csv")
insert_query = "INSERT INTO employees (name, role, salary) VALUES (%s, %s, %s)"
cursor.executemany(insert_query, new_employees)
connection.commit()

print(f"Inserted {cursor.rowcount} new employees.")

# Give a raise to all Engineers
cursor.execute("UPDATE employees SET salary = salary * 1.10 WHERE role = 'Engineer'")
connection.commit()
print(f"Updated salaries for {cursor.rowcount} engineers.")

# Fetch a report
cursor.execute("SELECT role, AVG(salary) FROM employees GROUP BY role")
for (role, avg_salary) in cursor:
print(f"Role: {role}, Average Salary: {avg_salary}")

except Error as e:
print(f"Error: {e}")
finally:
if connection.is_connected():
cursor.close()
connection.close()
print("Connection closed.")

Explanation:

We connect once at the start.
We ensure the table exists and then batch-insert employee data from a CSV file.
We run an UPDATE statement to modify salaries for a specific role.
We run a SELECT query to generate a summary report.
We handle errors and close the connection to release resources.

Conclusion

Interacting with MySQL in Python involves:

Establishing a secure, stable connection.
Using cursors to execute parameterized SQL queries.
Committing transactions to persist changes.
Handling exceptions and errors gracefully.
Employing best practices such as secure credential management, parameterization to prevent SQL injection, and using connection pooling for performance.

By understanding these concepts, you can confidently build Python applications that read, write, and manipulate MySQL data securely and efficiently.

Llama.cpp with Python

December 10, 2024December 10, 2024 ~ Davis Miller ~ Leave a comment

Introduction

Llama.cpp is an open-source project created by Georgi Gerganov and the community around large language models (LLMs). It provides an efficient and portable implementation to run LLaMA and other LLMs directly on CPU with minimal dependencies. While originally written in C++, llama.cpp has become very popular due to its ability to run models on commodity hardware, including laptops, and has inspired many bindings and integrations for different programming languages—most notably Python. Python, being a versatile and widely-used language in machine learning, has proven itself an excellent fit for orchestrating, managing, and extending LLaMA-based models. The interplay between llama.cpp and Python is a frequent topic of interest because it allows developers and researchers to leverage the computational efficiency of a low-level C++ library from the comfortable, high-level environment of Python.

This discussion will dive into the following aspects:

What is llama.cpp?
Core Features of llama.cpp
How llama.cpp Executes LLaMA and Similar Models
Why Integrate llama.cpp With Python?
Methods of Integrating llama.cpp With Python
Using the Official Python Bindings
Using Third-Party Python Wrappers or Libraries
Performance Considerations and Optimizations
Memory and Hardware Requirements
Advanced Topics: Prompting, Context Management, and Extensions
Examples of Python Code Interacting With llama.cpp
Future Directions and Considerations

1. What is llama.cpp?

Llama.cpp is a lightweight C++ implementation designed primarily to run Meta's LLaMA model and variants entirely on CPU with minimal resource usage. It relies on GGML (a library also written by Gerganov) for efficient low-level tensor operations, quantization routines, and inference optimizations specific to large language models. Initially, it began as a tool for running LLaMA on local machines without GPU acceleration, thus democratizing access to large language models. Over time, it expanded to support other models like Alpaca, Vicuna, MPT, and others that share a similar transformer architecture and parameter format.

2. Core Features of llama.cpp

CPU-Only Inference: It can run models without the need for expensive GPU hardware, relying on CPU acceleration and quantization to fit models into smaller memory footprints.
Quantization Support: Supports multiple quantization schemes (e.g., Q4_0, Q4_1, Q5_0, Q8_0) to reduce model size and memory usage without drastically sacrificing quality.
Portability: With minimal dependencies, it can be compiled and run on various systems, including Linux, macOS, and Windows, and even on ARM-based devices and smartphones.
Command-Line Interface: By default, llama.cpp provides a CLI tool for prompt input and interactive sessions.
Customization and Extensibility: Due to its open-source nature, developers can tweak it for more advanced or specialized uses.

3. How llama.cpp Executes LLaMA and Similar Models

Llama.cpp uses a transformer-based inference pipeline:

Model Loading: Reads a model's weights (often provided in a quantized format like .bin files). Weights are loaded into memory along with the necessary model parameters (layer counts, embedding size, etc.).
Tokenization: Converts input text strings into integer tokens using the same tokenizer as the original LLaMA model.
Forward Pass: Runs the tokens through a series of transformer blocks, computing intermediate states and probabilities for the next token.
Decoding: Chooses the next token based on the probability distribution (sampling, greedy, top-k, top-p strategies). This process repeats until completion.

Because llama.cpp is written in C++ and heavily optimized at a low level, it can handle these operations efficiently.

4. Why Integrate llama.cpp With Python?

Although llama.cpp provides a CLI, many developers prefer Python for automation, experimentation, and integration into larger machine learning pipelines. Reasons include:

Familiarity and Ecosystem: Python is the lingua franca of the machine learning community, featuring rich ecosystems like NumPy, PyTorch, TensorFlow, and Hugging Face's Transformers.
Rapid Prototyping: It's easy to write scripts, notebooks, and experiments in Python.
Integration and Deployment: Python makes it easy to integrate llama.cpp with web frameworks (e.g., Flask, FastAPI), automation tools, or other ML tools.
Data Handling and Pre-Processing: Python excels at handling data, making it simple to preprocess text or manage prompt templates before feeding them to the model.

5. Methods of Integrating llama.cpp With Python

There are typically two main ways to integrate llama.cpp with Python:

Direct Python Bindings (Cython or ctypes):
Compiling and exposing the C++ library functions directly to Python. This could involve using the Python's ctypes library or Cython, which bridges Python and C/C++ code seamlessly.
Command-Line Invocation from Python:
Using Python's subprocess module to call the llama.cpp CLI. This approach is simpler but less efficient, as it requires running a separate process and passing data via standard input/output.

Over time, official and community-driven efforts have produced native Python bindings and packages that simplify using llama.cpp directly from Python without dealing with low-level details.

6. Using the Official Python Bindings

As of late 2023 and early 2024, llama.cpp maintains a set of Python bindings within the repository:

Installation:
After cloning the llama.cpp repository, a pip install . command inside the python directory can build and install the Python package. Dependencies like cmake, a C++ compiler, and python3-dev (on Linux) may be required.

Usage:
Once installed, you can import llama_cpp in Python:

from llama_cpp import Llama

llm = Llama(model_path="/path/to/ggml-model-q4_0.bin")
output = llm("Hello, how are you?")
print(output)

Under the hood, the Python wrapper manages the lifetime of the model context, handles tokenization, inference, and decoding, providing a user-friendly interface.
Configuration Options: The Python bindings allow setting various inference parameters such as max_tokens, temperature, top_p, top_k, repeat_penalty, and others directly through function arguments.

7. Using Third-Party Python Wrappers or Libraries

In addition to the official bindings, community-driven projects integrate llama.cpp with Python-based LLM frameworks. Some examples include:

gpt4all: A Python package that uses llama.cpp under the hood to run local GPT-like models. It offers a Pythonic interface, model downloading, and a user-friendly API.
LangChain Integrations:
LangChain, a popular framework for building applications using LLMs, provides wrappers around llama.cpp models. This integration lets developers quickly integrate llama.cpp models into complex prompt chains, retrieval augmented generation (RAG) pipelines, and agent-based applications.
Hugging Face Transformers Integration:
While not officially integrated due to differing model formats and dependencies, the community has developed scripts that wrap llama.cpp inference into a transformers-like API, making it easier to switch between local CPU-based inference and standard huggingface model pipelines.

8. Performance Considerations and Optimizations

Running llama.cpp from Python introduces some performance trade-offs:

Overhead of Python Wrappers:
Direct C++ calls are faster than going through Python. The official bindings attempt to minimize overhead by doing bulk operations in C++ and exposing a clean Python API.
Batch Processing and Streaming:
To optimize performance, some Python wrappers offer streaming capabilities, where tokens are generated incrementally and fed to Python callbacks, reducing latency and memory overhead.
Quantization and Model Size Choices:
Choosing a quantized model (e.g., Q4_0 or Q4_K) allows running bigger models in limited memory. Performance and latency depend on how well quantization is applied and how large the context window is.
System Hardware and Compiler Optimizations:
Compiling llama.cpp with AVX2, AVX512, or NEON instructions (depending on CPU architecture) can drastically speed up inference. Python inherits these performance gains since it calls the optimized C++ backend.

9. Memory and Hardware Requirements

Model Size:
LLaMA models range from 7B to 65B parameters. Smaller models can be quantized down to a few GB of RAM usage, making them feasible on standard laptops. Larger models require more RAM.
Context Window and Prompt Size:
Increasing the context window increases memory usage. Python scripts must manage prompts and context to avoid exhausting system resources.
CPU Cores and Parallelization:
llama.cpp can utilize multiple CPU cores/threads. Python users can set the n_threads argument to spread computation across multiple CPU cores.

10. Advanced Topics: Prompting, Context Management, and Extensions

When integrating llama.cpp with Python, developers often need to handle:

System vs. User Prompts:
You can structure input to the model as a combination of system messages, user queries, and model responses, similar to the OpenAI ChatCompletion API. This helps maintain conversation state.
Context Windows:
Managing context so the model "remembers" previous user turns is a matter of feeding the entire conversation history each time. Python makes it easy to store and manipulate this state.
Plugins and Modules:
Python's flexibility allows extending the model's capabilities, for example:
- Integrating retrieval augmented generation by fetching context from a database or vector store.
- Using tools and APIs mid-conversation (e.g., calling external APIs when asked by the user).
Fine-Tuning and LoRA Adapters: While llama.cpp initially focused on inference, it has since supported LoRA-based fine-tuning. Python scripts can orchestrate the fine-tuning process and apply LoRA adapters to change model behavior.

11. Examples of Python Code Interacting With llama.cpp

Basic Prompting:

from llama_cpp import Llama

# Initialize the LLaMA model
llm = Llama(model_path="/path/to/llama-7b-q4.bin")

# Simple completion
result = llm(prompt="Explain the concept of gravity in simple terms.", max_tokens=128)
print(result['choices'][0]['text'])

Streaming Results for a Chat-Like Interface:

from llama_cpp import Llama

llm = Llama(model_path="/path/to/llama-13b-q4.bin")

# Stream tokens as they are generated
for token in llm(prompt="Write a poem about the sunrise", max_tokens=50, stream=True):
print(token['token'], end="", flush=True)

Integrating With LangChain:

from langchain.llms import LlamaCpp

llm = LlamaCpp(model_path="/path/to/llama-7b-q4.bin")
answer = llm("What is the capital of France?")
print(answer)

12. Future Directions and Considerations

GPU and Accelerator Support:
While llama.cpp is CPU-focused, future improvements or forks may incorporate GPU or specialized hardware acceleration. Python integrations would follow suit.
Improved Pythonic APIs: Over time, we may see more Python packages streamline the user experience, adding features like automatic model downloading, environment configuration, and integration with larger Python ML ecosystems.
Compatibility With More Model Families: As llama.cpp matures, it can potentially support a broader range of models (like Mistral or Falcon), making Python a universal interface to run various quantized LLMs locally.
Ecosystem Growth: With a growing user base, more utilities, templates, best practices, and tutorials will emerge, simplifying the integration process for newcomers.

Conclusion

llama.cpp has significantly lowered the barrier to entry for running large language models locally on CPUs. When integrated with Python, it becomes a powerful tool in a developer's or researcher's toolkit, combining the efficiency and low-level control of a C++ backend with the flexibility, rapid prototyping ability, and ecosystem integration of Python. With ongoing improvements, better bindings, and richer community support, the synergy between llama.cpp and Python will continue to evolve, delivering ever more capable and convenient local LLM solutions.

Python requests library

December 8, 2024December 9, 2024 ~ Davis Miller ~ Leave a comment

The Python requests library is one of the most popular and user-friendly HTTP libraries available for Python developers. It abstracts the complexities of making HTTP requests behind a simple and intuitive API, allowing you to interact with web services, APIs, and websites with ease. This comprehensive guide will delve into the requests library in great detail, covering installation, core features, advanced usage, best practices, and more.

Installation

Before using the requests library, you need to install it. It can be installed via pip:

pip install requests

To verify the installation and check the version:

import requests
print(requests.__version__)

As of the knowledge cutoff in October 2023, the latest stable version is typically recommended.

Basic Usage

The requests library simplifies HTTP requests by providing a straightforward API.

Making GET Requests

A GET request is used to retrieve data from a specified resource.

import requests

response = requests.get('api.example.com/data')
print(response.status_code) # HTTP status code
print(response.text) # Response body as text

Making POST Requests

A POST request is used to send data to a server to create/update a resource.

import requests

payload = {'key1': 'value1', 'key2': 'value2'}
response = requests.post('api.example.com/data', data=payload)
print(response.status_code)
print(response.json()) # If response is JSON

Request Parameters

When making HTTP requests, you often need to pass additional data like headers, query parameters, form data, etc. The requests library provides several parameters to facilitate this.

URL Parameters (params)

Used to send query parameters in the URL.

import requests

params = {'search': 'python', 'page': 2}
response = requests.get('api.example.com/search', params=params)
print(response.url) # api.example.com/search?search=python&page=2

Headers (headers)

Used to send custom HTTP headers with the request.

import requests

headers = {'Authorization': 'Bearer YOUR_TOKEN'}
response = requests.get('api.example.com/protected', headers=headers)

Form Data (data)

Used to send form-encoded data, typically for POST requests.

import requests

data = {'username': 'user', 'password': 'pass'}
response = requests.post('api.example.com/login', data=data)

JSON Payloads (json)

Used to send JSON-encoded data, automatically setting the Content-Type header to application/json.

import requests

json_data = {'key1': 'value1', 'key2': 'value2'}
response = requests.post('api.example.com/json', json=json_data)

Files (files)

Used to upload files.

import requests

files = {'file': open('report.pdf', 'rb')}
response = requests.post('api.example.com/upload', files=files)

For multiple files or additional form data, combine files with data.

import requests

files = {
'file1': open('report.pdf', 'rb'),
'file2': open('image.png', 'rb')
}
data = {'description': 'Report and image files'}
response = requests.post('api.example.com/upload', files=files, data=data)

Response Handling

After making a request, the requests library returns a Response object containing the server's response.

Status Codes

HTTP status codes indicate the result of the request.

import requests

response = requests.get('api.example.com/data')
print(response.status_code) # e.g., 200
print(response.ok) # True if status_code < 400

Common status codes:

200 OK: The request was successful.
201 Created: The resource was created successfully.
400 Bad Request: The server could not understand the request.
401 Unauthorized: Authentication is required.
403 Forbidden: The server understood the request but refuses to authorize it.
404 Not Found: The requested resource could not be found.
500 Internal Server Error: The server encountered an error.

Response Headers

Access response headers using the .headers attribute, which behaves like a case-insensitive dictionary.

import requests

response = requests.get('api.example.com/data')
print(response.headers['Content-Type'])

Response Body

The response body can be accessed in various formats.

Text

The .text attribute returns the response body as a string, using the response's encoding.

import requests

response = requests.get('api.example.com/data')
print(response.text)

JSON

If the response contains JSON, use the .json() method to parse it into Python data structures.

import requests

response = requests.get('api.example.com/data')
data = response.json()
print(data)

Note: If the response is not valid JSON, .json() will raise a ValueError.

Content

The .content attribute returns the response body in bytes, useful for binary data like images.

import requests

response = requests.get('api.example.com/image.png')
with open('image.png', 'wb') as f:
f.write(response.content)

Advanced Features

Sessions

Sessions allow you to persist certain parameters across multiple requests, such as cookies, headers, and authentication.

import requests

session = requests.Session()
session.headers.update({'Authorization': 'Bearer YOUR_TOKEN'})

# All requests using the session will include the Authorization header
response = session.get('api.example.com/protected')

Benefits of Sessions:

Connection Persistence: Reuses TCP connections, improving performance.
Session-wide Settings: Set default headers, authentication, etc.

Authentication

requests supports various authentication mechanisms.

Basic Authentication

import requests
from requests.auth import HTTPBasicAuth

response = requests.get(
'api.example.com/user',
auth=HTTPBasicAuth('username', 'password')
)

Alternatively, use a tuple:

response = requests.get(
'api.example.com/user',
auth=('username', 'password')
)

OAuth and Other Schemes

For more complex authentication schemes like OAuth, you may need to use external libraries or handle token acquisition manually.

SSL Verification

By default, requests verifies SSL certificates. You can disable it (not recommended) or provide a custom certificate.

# Disable SSL verification
response = requests.get('self-signed.example.com', verify=False)

# Provide a custom certificate
response = requests.get('secure.example.com', verify='/path/to/cert.pem')

Note: Disabling SSL verification can expose you to security risks like man-in-the-middle attacks.

Timeouts

Set timeouts to prevent your application from hanging indefinitely.

import requests

# Set a timeout of 5 seconds for the entire request
response = requests.get('api.example.com/data', timeout=5)

# Set connect and read timeouts separately
response = requests.get('api.example.com/data', timeout=(3, 7))

Exceptions:

requests.exceptions.Timeout: Raised when a timeout occurs.

Proxies

Route your requests through proxies.

import requests

proxies = {
'http': '10.10.1.10:3128',
'https': '10.10.1.10:1080',
}

response = requests.get('api.example.com/data', proxies=proxies)

For proxies requiring authentication:

proxies = {
'http': 'user:pass@10.10.1.10:3128/',
'https': 'user:pass@10.10.1.10:1080/',
}

response = requests.get('api.example.com/data', proxies=proxies)

Streaming Responses

Stream large responses to avoid loading them entirely into memory.

import requests

response = requests.get('api.example.com/largefile', stream=True)

with open('largefile', 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
if chunk:
f.write(chunk)

Redirection Control

By default, requests follows redirects. You can control this behavior.

import requests

# Do not follow redirects
response = requests.get('api.example.com/redirect', allow_redirects=False)

# Limit the number of redirects
response = requests.get('api.example.com/redirect', max_redirects=3)

Hooks

Hooks allow you to execute custom code at certain points during the request lifecycle.

import requests

def print_url(response, *args, **kwargs):
print(response.url)

hooks = {'response': print_url}

response = requests.get('api.example.com/data', hooks=hooks)

Exception Handling

Proper exception handling ensures your application can gracefully handle errors.

import requests
from requests.exceptions import HTTPError, Timeout, RequestException

try:
response = requests.get('api.example.com/data', timeout=5)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Timeout as timeout_err:
print(f'Timeout error occurred: {timeout_err}')
except RequestException as req_err:
print(f'An error occurred: {req_err}')
else:
print('Success!')

Common Exceptions:

requests.exceptions.HTTPError: Invalid HTTP response.
requests.exceptions.ConnectionError: Network problem (e.g., DNS failure, refused connection).
requests.exceptions.Timeout: Request timed out.
requests.exceptions.TooManyRedirects: Exceeded the configured number of maximum redirects.
requests.exceptions.RequestException: Base class for all exceptions.

Best Practices

Use Sessions: Reuse sessions to persist parameters and improve performance.

session = requests.Session()
session.headers.update({'User-Agent': 'my-app/0.0.1'})
response = session.get('api.example.com/data')

Handle Exceptions: Always anticipate and handle potential errors.

Set Timeouts: Prevent your application from hanging due to unresponsive servers.

Verify SSL Certificates: Ensure secure communication by verifying SSL certificates.

Limit Redirections: Control the number of redirects to prevent infinite loops.

Use Environment Variables for Sensitive Data: Store tokens and credentials securely.

import os
token = os.getenv('API_TOKEN')
headers = {'Authorization': f'Bearer {token}'}

Respect Rate Limits: When interacting with APIs, adhere to their rate limiting policies to avoid being blocked.

Use Streaming for Large Responses: Avoid loading large files into memory.

Clean Up Resources: Close files and other resources appropriately.

with open('file.txt', 'rb') as f:
response = requests.post('api.example.com/upload', files={'file': f})

Logging: Implement logging to monitor requests and responses, especially for debugging.

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

response = requests.get('api.example.com/data')
logger.info(f'Status Code: {response.status_code}')

Extending Requests

The requests library is highly extensible. You can create custom adapters, middleware, or even monkey-patch behaviors if needed.

Custom Adapters

Adapters allow you to define custom connection logic.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(
total=5,
backoff_factor=1,
status_forcelist=[500, 502, 503, 504],
method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('https://', adapter)
session.mount('http://', adapter)

response = session.get('api.example.com/data')

Middleware and Plugins

While requests does not have built-in middleware support, you can implement similar patterns using hooks or by subclassing.

For advanced use cases, consider using third-party libraries like requests-toolbelt which provide additional utilities.

Examples

Example 1: Downloading an Image

import requests

url = 'www.example.com/image.jpg'
response = requests.get(url, stream=True)

if response.status_code == 200:
with open('image.jpg', 'wb') as f:
for chunk in response.iter_content(1024):
f.write(chunk)
else:
print(f'Failed to retrieve image. Status code: {response.status_code}')

Example 2: Sending JSON Data with Authentication

import requests

url = 'api.example.com/create'
payload = {'name': 'John', 'age': 30}
headers = {'Authorization': 'Bearer YOUR_TOKEN'}

response = requests.post(url, json=payload, headers=headers)

if response.ok:
print('Resource created:', response.json())
else:
print(f'Error: {response.status_code} – {response.text}')

Example 3: Handling Pagination

Suppose an API uses pagination with page and per_page parameters.

import requests

url = 'api.example.com/items'
params = {'page': 1, 'per_page': 50}

all_items = []

while True:
response = requests.get(url, params=params)
if response.status_code != 200:
break
data = response.json()
items = data.get('items', [])
if not items:
break
all_items.extend(items)
params['page'] += 1

print(f'Total items retrieved: {len(all_items)}')

Example 4: Using Sessions with Retries

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry_strategy = Retry(
total=3,
status_forcelist=[429, 500, 502, 503, 504],
method_whitelist=["GET", "POST"],
backoff_factor=1
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount('https://', adapter)
session.mount('http://', adapter)

response = session.get('api.example.com/data')

Conclusion

The Python requests library is a powerful tool for making HTTP requests with ease and flexibility. Its simple API abstracts away many of the complexities involved in handling HTTP communications, making it accessible for beginners while still providing the depth needed for advanced use cases. By understanding and leveraging its features—such as sessions, authentication, SSL verification, and more—you can build robust applications that interact seamlessly with web services and APIs.

Whether you're scraping websites, consuming RESTful APIs, uploading files, or handling complex authentication schemes, the requests library provides the necessary tools to accomplish these tasks efficiently. Remember to follow best practices like handling exceptions, setting timeouts, and managing resources to ensure your applications are reliable and secure.