In the last article, we introduced the theory behind Recurrent Neural Networks. This time, we will use a hands-on example to illustrate the process of training a Vanilla RNN to forecast time series data.

To ensure our model actually learns a pattern (rather than just guessing yesterday’s value), we are going to use a cyclical dataset. We will generate a synthetic sine wave with a bit of random noise. This mimics real-world seasonal data and is the perfect playground for an RNN.

💎Full code at the end!💎

🛠️ DIY: Build your Vanilla RNN to forecast Time Series data

Theory is essential, but code is where the magic happens. To build our forecasting model, we’ll use Keras for the neural network architecture and Scikit-Learn to handle the data preprocessing. This combination allows us to focus on the RNN logic without getting bogged down in manual matrix multiplication.

Let’s start by importing our essential Data Science libraries:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error

1. Generating and Preparing the Data

Instead of downloading an external dataset, we will generate our own noisy wave. This guarantees you can reproduce these exact results.

# Generate 1000 data points of a sine wave with some random noise
time_steps = np.arange(0, 100, 0.1)
data = np.sin(time_steps) + np.random.normal(scale=0.1, size=len(time_steps))

# Put it into a pandas DataFrame
df = pd.DataFrame(data, columns=['Value'])

# Plot the first 200 points to see our pattern
plt.figure(figsize=(10, 4))
plt.plot(df['Value'][:200])
plt.title("Noisy Cyclical Data")
plt.show()

This plot helps verify that our data generation worked. You'll see a clear, repeating wave pattern, but with jagged, irregular edges caused by the np.random.normal noise we injected.

Scaling the Data

Neural Networks are highly sensitive to the scale of the input data. Since we will be using the tanh activation function (which outputs values between -1 and 1), it is best practice to scale our data. Here, we will scale everything to be strictly between 0 and 1.

# Initialize the scaler
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(df)

🎓 Agentic AI for Production

Most developers hit the exact same wall with Agentic AI: it works beautifully on their local machine, and then completely falls apart in production.

If you’re ready to escape “demo purgatory,” check out Your Path to Agentic AI for Production* by Towards AI and Paul Iusztin.

This isn’t a passive, follow-along tutorial. It’s a hands-on sprint to teach you the system design fundamentals that outlast today’s trending frameworks. You will leave with two fully deployed systems for your portfolio:

• An Autonomous Research Agent: Master multi-source data collection, ReAct reasoning loops, and tool orchestration (using Gemini, Perplexity, and Firecrawl).

• A Multi-Modal Writing Workflow: Implement evaluator-optimizer patterns and LangGraph orchestration to turn research into publication-ready content.

It includes:

• Lifetime access

• Active engineering Discord

• Live introductory calls.

Enroll Now!

*by purchasing any of their courses you would also be supporting MLPills.

2. Reframing as a Supervised Learning Problem

RNNs cannot just ingest a long list of numbers. We need to create a “sliding window” of data. We will use a sequence of previous steps (e.g., the last 20 points) to predict the very next step.

# Initialize empty lists
X = []
y = []

# Set the number of historical steps the model will look at
timesteps = 20

# Iterate to populate the lists
for i in range(len(scaled_data) - timesteps):
    X.append(scaled_data[i : i + timesteps, 0])
    y.append(scaled_data[i + timesteps, 0])

# Convert lists to numpy arrays
X = np.array(X)
y = np.array(y)

Neural Networks employ a unique input structure. Specifically, for RNNs, the dimensions of the input tensor must be exactly three-dimensional: [number of samples, number of timesteps, number of features].

Currently, our X array is two-dimensional. We only have one feature (the value itself), so we need to reshape our array to explicitly include that feature dimension.

# Reshape X to [samples, timesteps, features]
X = X.reshape(X.shape[0], X.shape[1], 1)

print(f"X shape: {X.shape}")
print(f"y shape: {y.shape}")

Output:

X shape: (980, 20, 1), y shape: (980,)

We started with 1000 data points. Because our sliding window requires 20 historical points to make its first prediction, we "lose" the first 20 steps, leaving us with 980 usable samples. The 20 represents the historical timesteps our model looks at for each sample. The 1 is our single feature (the value of the sine wave). The y array naturally has 980 corresponding target values.

3. Train/Test Split

Let’s select 80% of the data for training and the remaining 20% to test our model’s performance. Crucial note: Because this is time-series data, we must split it sequentially. We cannot randomly shuffle the data, or we will leak the future into the past!

split_idx = int(X.shape[0] * 0.8)

X_train, X_test = X[:split_idx], X[split_idx:]
y_train, y_test = y[:split_idx], y[split_idx:]

4. Building the Vanilla RNN Model

First, import the required Keras modules:

from keras.models import Sequential
from keras.layers import SimpleRNN, Dense

Let’s build the model. We will use a single SimpleRNN layer followed by a Dense layer to output our single prediction.

• Units: We’ll use 16 neurons. Since our dataset has a clear pattern, we don’t need a massive network.

• Activation: We use tanh, which is standard for RNNs as it helps mitigate vanishing gradients better than sigmoid.

• Dropout: We are intentionally skipping the Dropout layer here. While dropout is great for complex, noisy data (like stocks) to prevent overfitting, applying it to a simple, deterministic pattern often just introduces artificial noise and prevents the model from learning the curve smoothly.

# Initialize the Sequential model
model = Sequential()

# Add an RNN layer with 16 units
model.add(SimpleRNN(16, 
                    activation='tanh', 
                    input_shape=(X_train.shape[1], 1)))

# Add a Dense layer with 1 unit to output the final prediction
model.add(Dense(1))

# Compile the model using Mean Squared Error and the Adam optimizer
model.compile(loss='mean_squared_error', optimizer='adam')

model.summary()

Output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ simple_rnn (SimpleRNN)          │ (None, 16)             │           288 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 1)              │            17 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

The (None, 16) output shape means the layer outputs 16 values (our 16 units), while None is just a placeholder representing a variable batch size. The 288 parameters in the RNN layer come from three sets of weights: the input-to-hidden weights (1 feature × 16 units = 16), the hidden-to-hidden recurrent weights (16 units × 16 units = 256), and the biases (16). Added together, 16 + 256 + 16 = 288. Finally, the Dense layer has 17 parameters (16 incoming weights from the RNN + 1 bias).

5. Training the Model

We will train the model for 30 epochs. We will also use a validation_split to monitor how well the model generalizes to unseen data during training.

# Fit the model
history = model.fit(X_train, y_train, 
                    epochs=30, 
                    batch_size=16, 
                    validation_split=0.1, 
                    shuffle=False, # Remember, no shuffling in time series!
                    verbose=1)

Let’s visualize the training process to ensure our loss decreased smoothly:

plt.figure(figsize=(8, 4))
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss Over Time')
plt.legend()
plt.show()

A line graph with two downward-trending curves. The 'Training Loss' (how well the model fits the training data) and 'Validation Loss' (how well it performs on the 10% hold-out set) should both drop rapidly in the first few epochs and then smoothly level out. Because they track closely together without the validation loss spiking back up, it confirms our model is learning successfully without overfitting.

6. Evaluating the Results

Now for the moment of truth. Let’s make predictions on our test set and see if the model successfully learned the cyclical pattern.

# Generate predictions
y_pred = model.predict(X_test)

# Inverse transform to get the values back to their original scale
y_test_scaled_back = scaler.inverse_transform(y_test.reshape(-1, 1))
y_pred_scaled_back = scaler.inverse_transform(y_pred)

# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test_scaled_back, y_pred_scaled_back))
print(f"Test RMSE: {rmse:.4f}")

Output:

Test RMSE: 0.1221

RMSE (Root Mean Squared Error) measures the average magnitude of our model's errors. An RMSE of 0.1221 on a dataset that naturally fluctuates between roughly -1.2 and 1.2 is very low. It tells us that our model's predictions are, on average, only off by about 0.12 units from the true values.

You should see a very low RMSE score. To truly appreciate the results, let’s plot the actual values versus the model’s predictions.

plt.figure(figsize=(12, 5))
plt.plot(y_test_scaled_back, label='Actual Data', color='blue')
plt.plot(y_pred_scaled_back, label='RNN Predictions', color='red', linestyle='--')
plt.title('Vanilla RNN: Actual vs Predicted')
plt.legend()
plt.show()

Your plot shows the red dashed line (predictions) hugging the blue line (actual data) almost perfectly. The Vanilla RNN has successfully learned the underlying sequential logic of the dataset! Notice how the model doesn't just blindly copy the previous step; it actively anticipates the curve's peaks and valleys, demonstrating that it has captured the true temporal pattern.

As an 💎Extra Issue💎, next Wednesday we will share more details about this analysis (only for paid subscribers).

Here you have the full code as promised:

Read more

The nurse assesses you in seconds: “Chest pain? Go to the ER immediately.” “Sprained ankle? Go to Urgent Care.” “Just a check-up? Go to Family Medicine.”

This triage process saves lives and resources. In the world of AI Agents, this is the Routing pattern.

In this issue we will:

• Define the Routing pattern and why “one prompt fits all” is a myth.

• Explore how to use Structured Outputs to make reliable decisions.

• Build a “Customer Support Triage” agent that routes queries to billing, technical support, or sales specialists. 💎 With all the code + notebook! 💎

Let’s begin!

💊 Pill of the week

This is the essence of the Routing pattern. In simple applications, we often stuff a single prompt with instructions for every possible scenario: “If the user asks about X, do Y. If they ask about A, do B...”

As your application grows, this “Mega-Prompt” becomes a nightmare. It becomes slow, expensive, and confusing for the LLM.

Routing solves this by introducing a preliminary step. The LLM acts as a traffic controller. It classifies the input intent and then directs the flow to a specialized handler (which could be another prompt, a tool, or even a completely different model).

In previous articles, we covered the Orchestrator-Worker: Break a complex task into subtasks and do them all (e.g., “Research, then Write, then Format”).

Routing is different. It is about Selection. It is “Either/Or”, not “And/Then”.

Why is Routing so Effective?

1. Specialization: You can have a prompt specifically tuned for “Writing SQL” and another for “Writing Poems.” They don’t need to know about each other.

2. Cost & Speed: You can use a small, fast model (like GPT-4o-mini or a local Llama model) to do the routing, and only call the expensive “Smart” model when the task actually requires it.

3. Safety: You can route “sensitive” topics to a hard-coded refusal response without ever engaging the generative engine.

There are other patterns that we’ve already covered, here you can get a summary of them:

📖 Book of the Week

I highly recommend checking out the new release, AI-Native LLM Security: Threats, Defenses, and Best Practices for Building Safe and Trustworthy AI.

What makes this book stand out is the pedigree of its authors—Vaibhav Malik, Ken Huang, and Ads Dawson are pioneers involved in the OWASP Top 10 for LLM applications. They aren’t just theorizing; they are the ones defining the standards.

What You’ll Gain: Rather than offering vague advice on “being careful,” this guide provides a technical blueprint for:

• Secure-by-Design Architecture: Strategies for isolation and access control that are baked in from day one, not bolted on later.

• Operationalizing Security (MLSecOps): How to integrate security controls directly into your CI/CD and MLOps pipelines.

• Framework Integration: Practical ways to leverage established taxonomies from OWASP, NIST, and MITRE to identify vulnerabilities.

• The Full Lifecycle: Mitigating risk from data curation all the way to incident response in live deployments.

Whether you are navigating the complex legal landscape of AI or trying to prevent prompt injections, this is the comprehensive manual the industry has been waiting for.

Get the book

🛠️ DIY: The Customer Support Triage Bot

Imagine you are building a support bot for a SaaS company. You receive three types of emails:

1. Technical: “I can’t log in,” “The API is throwing a 500 error.”

2. Billing: “I want a refund,” “Update my credit card.”

3. General: “How are you?”, “What is your pricing?”

We don’t want our expensive “Technical Expert” model wasting tokens processing a refund request. We want to Route them.

The Challenge

We will build a workflow that:

1. Classifies an incoming user query into a distinct category.

2. Selects the appropriate “Expert Chain” based on that category.

3. Executes only the selected chain.

Setting the Stage

We will use LangChain’s structured output capabilities (Pydantic) to ensure our Router makes a firm decision, not a fuzzy text response.

# Install necessary libraries
# !pip install langchain-openai langchain pydantic

import os
from typing import Literal
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from pydantic import BaseModel, Field

# Set up your OpenAI API key
# os.environ["OPENAI_API_KEY"] = "your_api_key"

# Initialize our LLMs
# The router can be a faster model. The experts might be smarter models.
router_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) # Fast & Deterministic
expert_llm = ChatOpenAI(model="gpt-4.1", temperature=0.7)   # Creative & Capable

Step 1: Building the Router

The Router’s only job is to output a Category. We don’t want it to answer the question yet.

Read more

In this issue we will:

• Define the “Evaluator-Optimiser” pattern and its feedback loops.

• Explore how to implement a feedback cycle using LangChain.

• Build a step-by-step example that refines Python code until it is bug-free. 💎 With all the code + notebook! 💎

Let’s begin!

💊 Pill of the week

This is the essence of the Evaluator–Optimiser pattern. Most LLM interactions are “one-shot”—you ask, and it answers. But for high-stakes tasks, the first answer is rarely the best one.

In this workflow, we separate the “creative” process from the “critical” process. One LLM acts as the Generator (the writer), producing an initial attempt. A second LLM acts as the Evaluator (the editor), scoring the output against strict criteria and providing feedback. If the output isn’t good enough, the feedback is fed back into the Generator to improve the next version.

In a previous article, we covered the Orchestrator-Worker pattern, where work is split horizontally across different specialists:

The key difference? Orchestrator-Worker is about breadth (coordinating many different subtasks). Evaluator-Optimiser is about depth (iterating on a single task until quality is maximized).

Why is Evaluator-Optimiser so Effective?

• Self-Correction: LLMs often hallucinate or make small logic errors. An explicit evaluation step catches these before they reach the user.

• Separation of Concerns: It is hard for a single prompt to be both “creative” and “critical” at the same time. This pattern lets one model focus on generating ideas and another on strict quality control.

• Higher Quality Ceiling: By allowing multiple attempts, you move beyond the “average” probabilistic response to a highly refined output.

• Alignment: You can enforce specific business rules (e.g., “Must be under 280 chars”, “Must use JSON format”) more reliably by rejecting outputs that fail these checks.

This pattern is perfect for code generation, legal document drafting, or complex math problems where “almost right” is effectively “wrong.”

Under the Hood

The Evaluator-Optimiser pattern involves a circular flow:

1. The Generator: Produces the initial response or a revised response based on feedback.

2. The Evaluator: analyzes the response against acceptance criteria. It outputs a decision (PASS/FAIL) and structured feedback.

3. The Loop: If the result is a PASS (or max attempts reached), we exit. If FAIL, the feedback flows back to the Generator.

Let’s see this in action!

📖 Book of the Week

The AI Optimization Playbook by Dr. Chun Schiros, Supreet Kaur, Rajdeep Arora and Dr. Usha Jagannathan is a clear, execution-focused guide for leaders who want AI to deliver real business outcomes rather than endless pilots.

What will you learn?
How to design AI strategies that are explicitly aligned with business goals and ROI
How to select, prioritise, and justify AI initiatives in an enterprise context
How to move from proofs of concept to production using MLOps and LLMOps practices
How to measure success beyond model accuracy, including adoption, experimentation, and causal impact
How to operationalise generative AI and AI agents responsibly at scale
How to embed explainability, fairness, and regulatory compliance into AI systems from day one

Why should you read it?
It treats AI as an organisational capability, not just a technical one
The focus is on decision-making, governance, and scale, which is where most AI efforts struggle
Responsible AI is positioned as a strategic enabler, not a box-ticking exercise
Real-world case studies help bridge the gap between executives, product leaders, and technical teams
It avoids hype and tool obsession, favouring durable frameworks that hold up over time

If you are responsible for turning AI investment into measurable impact across an organisation, this book is a practical and credible companion. Highly recommended for CTOs, CIOs, CDAOs, and senior AI leaders who want results, not demos.

Get it here

🛠️ DIY: The Self-Healing Code Generator

Imagine you are building an AI coding assistant. You don’t want it to just generate code that looks right; you want code that is actually correct and follows best practices.

The Challenge

We will build a workflow that:

1. Takes a user request for a Python function.

2. Generates a candidate solution.

3. Evaluates the code (simulating a “senior dev” review) for correctness, efficiency, and style.

4. Optimizes the code based on the review until it passes.

As an example, we’ll ask it to: “Write a Python function to check if two strings are valid anagrams.”

Setting the Stage

We will use LangChain to manage our prompts and Pydantic to structure the critical “Evaluation” step.

# Install necessary libraries if you haven't already
# !pip install langchain-openai langchain pydantic

import os
from typing import List, Optional

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser, JsonOutputParser
from pydantic import BaseModel, Field

# Set up your OpenAI API key
# os.environ["OPENAI_API_KEY"] = "your_api_key"

# Initialize our LLMs
# We use a higher temperature for generation (creativity) 
# and a low temperature for evaluation (strictness).
generator_llm = ChatOpenAI(model="gpt-4.1", temperature=0.7)
evaluator_llm = ChatOpenAI(model="gpt-4.1", temperature=0.0)

This section initializes the tools we need.

• Imports: We import ChatOpenAI (to talk to the models), BaseModel/Field (to define strict data structures), and output parsers (to turn raw AI text into usable code objects).

• LLM Initialization (Crucial Detail):

  • generator_llm uses temperature=0.7. This adds “creativity” or randomness, helping the model try different approaches if it gets stuck.

  • evaluator_llm uses temperature=0.0. This makes the model deterministic and strict. We want the “critic” to be consistent, not creative.

Step 1: Building the Generator

The generator needs to handle two states: creating a fresh draft, and refining a draft based on feedback.

generator_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are an expert Python developer. Your goal is to write clean, efficient, and documented Python code.
    
    If you are provided with FEEDBACK, you must revise your previous code to address the specific issues mentioned.
    Do not explain your thought process in the output, just output the Python code block."""),
    ("user", """Request: {request}
    
    {context_placeholder}""")
])

def generator_chain(request, feedback=None, previous_code=None):
    context = ""
    if feedback and previous_code:
        context = f"\nPREVIOUS CODE:\n{previous_code}\n\nFEEDBACK TO ADDRESS:\n{feedback}"
    
    chain = generator_prompt.partial(context_placeholder=context) | generator_llm | StrOutputParser()
    return chain.invoke({"request": request})

This creates the “Writer” of our story.

• generator_prompt: This tells the LLM it is a Python expert. Crucially, it has a conditional instruction: “If you are provided with FEEDBACK, you must revise...” This allows the prompt to work for both the first draft (no feedback) and subsequent revisions.

• generator_chain function:

  • It accepts feedback and previous_code as optional arguments.

  • If feedback exists, it injects it into the prompt context so the LLM knows what to fix.

  • If no feedback exists (the first run), it just sees the original request.

Step 2: Building the Evaluator

The evaluator is the strict “Senior Engineer.” It must output a structured decision so our code knows whether to loop or stop.

# Define the evaluation structure
class Evaluation(BaseModel):
    decision: str = Field(description="The decision: 'PASS' if the code is perfect, 'NEEDS_IMPROVEMENT' otherwise.")
    feedback: str = Field(description="Specific feedback on what is wrong or how to improve. If PASS, say 'Looks good'.")

evaluator_parser = JsonOutputParser(pydantic_object=Evaluation)

evaluator_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a Senior Software Engineer conducting a code review. 
    Analyze the provided Python code for:
    1. Correctness (Does it solve the problem?)
    2. Efficiency (Is the time/space complexity optimal?)
    3. Style (Does it follow PEP8 and include docstrings?)
    
    Be strict. If the code is inefficient or lacks docstrings, mark it as NEEDS_IMPROVEMENT.
    
    {format_instructions}"""),
    ("user", """Original Request: {request}
    
    Code to Review:
    {code}""")
]).partial(format_instructions=evaluator_parser.get_format_instructions())

evaluator_chain = evaluator_prompt | evaluator_llm | evaluator_parser

This creates the “Editor” or “Critic.”

• class Evaluation(BaseModel): This is the most important part of this step. We force the LLM to return data in a specific JSON format containing exactly two fields:

  • decision: A “PASS” or “NEEDS_IMPROVEMENT” flag. We can use this in our Python if statements later.

  • feedback: Text explaining why it failed.

• evaluator_prompt: The system instructions tell the model to be “strict” and check for specific criteria (correctness, efficiency, style). This ensures it doesn’t just say “Good job!” to bad code.

Step 3: The Workflow Loop

Now we combine them into a while loop that runs until the evaluator is happy or we run out of attempts.

def evaluator_optimiser_workflow(request: str, max_attempts: int = 3):
    print(f"🚀 Starting task: {request}\n")
    
    current_code = ""
    feedback = ""
    attempt = 1
    
    while attempt <= max_attempts:
        print(f"--- Attempt {attempt}/{max_attempts} ---")
        
        # 1. GENERATE
        print("⚡ Generating code...")
        current_code = generator_chain(request, feedback, current_code)
        # Just a helper to print a snippet of the code
        print(f"   (Generated {len(current_code)} chars of code)")
        
        # 2. EVALUATE
        print("🧐 Evaluating...")
        evaluation = evaluator_chain.invoke({
            "request": request, 
            "code": current_code
        })
        
        decision = evaluation['decision']
        feedback = evaluation['feedback']
        
        print(f"   Decision: {decision}")
        print(f"   Feedback: {feedback}\n")
        
        # 3. CHECK EXIT CONDITION
        if decision == "PASS":
            print("✅ Quality Standard Met! Returning final code.")
            return current_code
            
        attempt += 1
        
    print("⚠️ Max attempts reached. Returning last version.")
    return current_code

This is the “Orchestration” logic that ties the two agents together.

• while attempt <= max_attempts: This creates a safety loop. We don’t want an infinite loop if the AI never gets it right, so we cap it (e.g., at 3 tries).

• The Execution Flow:

  1. Generate: It calls the Generator. On the first loop, feedback is empty.

  2. Evaluate: It takes the code we just made and sends it to the Evaluator.

  3. Parse: It extracts the decision and feedback from the JSON response.

  4. Decision Point:

     • If PASS: The function breaks the loop immediately and returns the code. The job is done.

     • If NEEDS_IMPROVEMENT: The loop restarts, but this time, the feedback variable is full. The Generator will see this feedback in the next iteration.

Putting It to the Test

Let’s run our “Anagram Checker” request. To see the pattern work, I will simulate a difficult request by asking for an optimized solution.

request = "Write a function `is_anagram(s1, s2)` that checks if two strings are anagrams. It must be O(n) time complexity and handle case-insensitivity."

final_code = evaluator_optimiser_workflow(request)

print("=" * 60)
print("FINAL RESULT:")
print(final_code)

This is simply the “Run” button.

• It defines a complex request (Anagram checker, O(n) complexity).

• It calls the main workflow function.

• It prints the final result after the loop has finished (either because it passed or because it ran out of tries).

Possible Output Flow:

🚀 Starting task: Write a function `is_anagram(s1, s2)`...

--- Attempt 1/3 ---
⚡ Generating code...
   (Generated 150 chars of code)
🧐 Evaluating...
   Decision: NEEDS_IMPROVEMENT
   Feedback: The solution uses sorting (sorted(s1) == sorted(s2)), which is O(n log n). The request specifically asked for O(n) time complexity. Please use a hash map or frequency counter array instead. Also, ensure docstrings are added.

--- Attempt 2/3 ---
⚡ Generating code...
   (Generated 320 chars of code)
🧐 Evaluating...
   Decision: NEEDS_IMPROVEMENT
   Feedback: Logic is now O(n) using Counter, which is good. However, you forgot to handle case-insensitivity as requested. "Listen" and "silent" should return True.

--- Attempt 3/3 ---
⚡ Generating code...
   (Generated 340 chars of code)
🧐 Evaluating...
   Decision: PASS
   Feedback: Looks good. Time complexity is O(n), logic handles case insensitivity correctly, and docstrings are present.

✅ Quality Standard Met! Returning final code.

============================================================
FINAL RESULT:
def is_anagram(s1: str, s2: str) -> bool:
    """
    Checks if two strings are anagrams of each other.
    Time Complexity: O(n)
    """
    from collections import Counter
    
    # Normalize strings: remove spaces and convert to lowercase
    s1 = s1.replace(" ", "").lower()
    s2 = s2.replace(" ", "").lower()
    
    if len(s1) != len(s2):
        return False
        
    return Counter(s1) == Counter(s2)

See what happened? The Evaluator caught the complexity issue (Attempt 1) and the functional bug (Attempt 2), forcing the Generator to fix them. A standard “one-shot” prompt might have settled for the easier sorted() solution.

Here you have a summary:

1. User Request → Generator → Draft Code

2. Draft Code + Requirements → Evaluator → Feedback + Decision

3. If Fail: Feedback → Generator → New Draft Code (Repeat)

4. If Pass: Return Final Code

Here you have all the code:

Play with the code

When to Reach for Evaluator-Optimiser?

The Evaluator-Optimiser pattern is your best choice when:

• Quality > Speed: You can afford the latency of multiple LLM calls to get a superior result.

• Objective Criteria Exists: It works best when success is measurable (e.g., code syntax, word count, JSON schema, absence of specific words).

• The “First Draft” is usually flabby: Tasks like coding, creative writing, or summarizing complex documents often benefit from a “review” phase.

⚠️ Important Considerations

1. Infinite Loops: Always set a max_attempts counter. If the generator can’t satisfy the evaluator, you don’t want to burn tokens forever.

2. The “Strictness” Balance: If the Evaluator is too strict or vague in its feedback, the Generator might get confused and degrade the output. The feedback prompt is just as important as the generation prompt.

3. Context Window: Passing the PREVIOUS CODE and FEEDBACK back into the generator grows the context. For very long tasks, you might need to manage history carefully.

Conclusion

The Evaluator-Optimiser pattern mimics the human workflow of “drafting and revising.” By accepting that the first output is rarely perfect, we build systems that are resilient, self-correcting, and capable of much higher quality than raw models alone.

Next time you are building an agent, ask yourself: Does this need to be fast, or does it need to be right? If the answer is “right,” hire an Evaluator.

Imagine you’re managing a complex research project. You need to analyze a company’s financial health, but that requires gathering quarterly reports, checking recent news, analyzing competitor data, and synthesizing market trends. No single person can efficiently do all of this at once—and asking them to do it sequentially would take forever.

This is where the Orchestrator-Worker pattern comes in. It’s a workflow pattern that transforms your AI from a solo performer into a coordinated team, with one “manager” LLM breaking down complex tasks and delegating specialized work to multiple “expert” workers.

Think of it like running a restaurant kitchen. The head chef (orchestrator) doesn’t cook every dish personally. Instead, they plan the menu, coordinate timing, and delegate: the grill station handles steaks, the pastry chef makes desserts, and the sous chef prepares sauces. Each specialist focuses on what they do best, and the head chef ensures everything comes together into a cohesive dining experience.

In the previous article we covered Parallelization, where independent predefined tasks run simultaneously:

The key difference? In Parallelization, all tasks are predetermined and run on the same input. In Orchestrator-Worker, the orchestrator dynamically decides what tasks to create, potentially with different inputs for each worker, and then synthesizes their diverse outputs into a final answer (which can also be run in parallel if wanted).

In this issue we will:

• Define the “Orchestrator-Worker” pattern and explain its key benefits

• Explore how to implement this pattern using LangChain

• Build a step-by-step example that analyzes a company comprehensively using specialized workers 💎 With all the code + notebook! 💎

This is one out of several other workflow patterns, which we covered here:

Quick note before we start: I’ll be sharing the LangGraph version of this workflow in parallel later this week, available only to 💎paid subscribers.

Subscribe now

Let’s begin with the Orchestrator-Worker pattern in LangChain!

Why is Orchestrator-Worker so Effective?

• Task Decomposition: The orchestrator intelligently breaks down complex queries into manageable subtasks, each handled by the most appropriate specialist.

• Specialization & Modularity: Each worker can use different models, prompts, tools, or knowledge bases optimized for their specific domain. You can easily add, remove, or replace workers without redesigning the entire system.

• Scalability: As your requirements grow, you simply add new specialized workers rather than making one massive, unwieldy prompt.

• Dynamic Adaptation: Unlike fixed patterns, the orchestrator can adjust its plan based on the query, skipping irrelevant subtasks or adding new ones as needed.

This pattern is perfect for scenarios where a complex question requires multiple types of expertise, and you need intelligent coordination rather than just running everything in a predetermined way in parallel.

Under the Hood

The Orchestrator-Worker pattern in LangChain involves three key components:

1. The Orchestrator: An LLM that analyzes the user’s request and creates a plan by breaking it into subtasks

2. The Workers: Specialized chains (potentially different LLMs, tools, or knowledge bases) that each handle one type of subtask

3. The Synthesizer: An LLM that takes all worker outputs and integrates them into a coherent final answer

The flow looks like this:

Let’s see this in action.

🛠️ DIY: Product Launch Analysis

Imagine you’re building an internal analysis tool for your company. When preparing for a product launch review, you need to provide a comprehensive analysis covering multiple dimensions: technical readiness, market positioning, and risk assessment—all from internal documentation and simulated data.

The Challenge

We’ll take a product name and use an orchestrator to:

1. Determine which aspects of the product need analysis

2. Delegate each aspect to a specialized worker:

   • Technical Readiness Worker: Analyzes features and development status

   • Market Position Worker: Evaluates competitive positioning and target audience

   • Risk Assessment Worker: Identifies potential challenges and mitigation strategies

3. Synthesize all findings into a comprehensive launch readiness report

As an example, we’ll analyze: “SmartHome Hub Pro”

Let’s begin!

Setting the Stage

First, we need our LangChain tools and our LLM. We’ll also import Pydantic to define structured outputs.

# Install necessary libraries if you haven’t already
# !pip install langchain-openai langchain pydantic

import os
from typing import List, Dict

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser, JsonOutputParser
from langchain_core.pydantic_v1 import BaseModel, Field

# Set up your OpenAI API key
# os.environ[”OPENAI_API_KEY”] = “your_api_key”

# Initialize our LLMs
orchestrator_llm = ChatOpenAI(model=”gpt-4o”, temperature=0)
worker_llm = ChatOpenAI(model=”gpt-4o-mini”, temperature=0.3)
synthesizer_llm = ChatOpenAI(model=”gpt-4o”, temperature=0.2)

• orchestrator_llm: Uses the more capable model for planning (temperature=0 for consistency)

• worker_llm: Uses a faster, cheaper model for specialized tasks

• synthesizer_llm: Uses the capable model again for final integration

Step 1: Building the Orchestrator

The orchestrator’s job is to analyze the query and create a structured plan of subtasks.

# Define the plan structure
class SubTask(BaseModel):
    task_id: str = Field(description=”Unique identifier for the subtask”)
    worker_type: str = Field(description=”Type of worker needed: ‘technical’, ‘market’, or ‘risk’”)
    instructions: str = Field(description=”Specific instructions for this subtask”)

class Plan(BaseModel):
    product: str = Field(description=”The product being analyzed”)
    subtasks: List[SubTask] = Field(description=”List of subtasks to execute”)

# Set up the orchestrator parser
orchestrator_parser = JsonOutputParser(pydantic_object=Plan)

# Create the orchestrator prompt
orchestrator_prompt = ChatPromptTemplate.from_messages([
    (”system”, “”“You are an expert product launch coordinator. Your job is to analyze a product launch request and break it down into specific subtasks for specialized workers.

Available workers:
- ‘technical’: Analyzes technical readiness, features, and development status
- ‘market’: Evaluates market positioning, target audience, and competitive landscape
- ‘risk’: Identifies potential risks and mitigation strategies

For the given product, create a comprehensive plan that covers all relevant aspects for launch readiness. Each subtask should have clear, specific instructions.”“”),
    (”user”, “Product to analyze: {product}\n\n{format_instructions}”)
]).partial(format_instructions=orchestrator_parser.get_format_instructions())

# Define the orchestrator chain
orchestrator_chain = orchestrator_prompt | orchestrator_llm | orchestrator_parser

• We define a structured Plan with multiple SubTask objects

• The orchestrator receives the product name and creates a JSON plan

• Each subtask specifies which worker type to use and what that worker should do. In this case we defined three different workers, which we will implement in the next step:

  • the technical specialist

  • the market specialist

  • the risk specialist

📖 Book of the Week

If you are building enterprise-grade LLM systems or are responsible for bringing GenAI into complex organisational environments, this is a standout strategic and practical guide.

“LLMs in Enterprise: Design strategies, patterns, and best practices for large language model development” by Ahmed Menshawy and Mahmoud Fahmy

💡 A comprehensive, end-to-end playbook for designing, optimising, deploying, and governing large language model applications at enterprise scale. You will move from foundational concepts through to advanced design patterns, fine-tuning approaches, deployment architectures, and real operational concerns such as evaluation, monitoring, compliance, and cost optimisation.

What sets it apart

It brings a deeply structured, pattern-driven perspective to enterprise LLM development, going far beyond simple demos or hobby-level guidance. You get clear strategies for real-world challenges: scaling across business units, improving reliability, integrating RAG, ensuring fairness and transparency, and managing production LLM systems responsibly.

Each chapter bridges theory with pragmatic guidance, giving teams a common language for building robust and future-proof GenAI applications.

You will learn to:

✅ Apply proven design patterns to integrate LLMs into enterprise systems
✅ Overcome challenges in scaling, deploying, and optimising LLM applications
✅ Use fine-tuning techniques, contextual customisation, and RAG to boost performance
✅ Build data strategies that genuinely improve LLM quality and reliability
✅ Implement advanced inferencing engines and performance optimisation patterns
✅ Evaluate LLM applications with enterprise-ready metrics and frameworks
✅ Monitor production LLMs and ensure security, privacy, and compliance
✅ Understand responsible AI practices, including transparency and robustness
✅ Track emerging trends, multimodality, and the next wave of GenAI capabilities

If you are serious about deploying LLMs across an organisation and want proven patterns that reduce risk and accelerate delivery, this is the resource that turns experimentation into scalable enterprise AI.

Get it here

Step 2: Building the Specialized Workers

Each worker is a specialized chain optimized for its domain, with access to relevant internal data (PRODUCT_DATABASE).

Worker 1: Technical Readiness Analyst

technical_prompt = ChatPromptTemplate.from_template(
    “”“You are a technical readiness expert. Analyze the following product based on these instructions:

Product: {product}
Instructions: {instructions}

Product Data:
Features: {features}
Development Status: {development_status}

Provide a concise but thorough technical readiness analysis. Assess completeness, quality indicators, and readiness for launch.”“”
)

def technical_worker_invoke(inputs):
    product = inputs[’product’]
    product_data = PRODUCT_DATABASE.get(product, {})
    
    return (technical_prompt | worker_llm | StrOutputParser()).invoke({
        “product”: product,
        “instructions”: inputs[’instructions’],
        “features”: product_data.get(’features’, []),
        “development_status”: product_data.get(’development_status’, {})
    })

Worker 2: Market Position Analyst

Read more

Imagine you’re a news editor trying to understand a breaking story. You get a single field report. To really cover it, you need to know the “who, what, where” (the key facts), the “so what” (the summary), and the “how do people feel” (the public sentiment) all at once. Asking an AI to do this sequentially—first find the facts, then write the summary, then analyze the sentiment—is slow and inefficient.

This is where Parallelization comes in. It’s a workflow pattern that transforms your AI from a single-track worker into a multi-talented team, tackling multiple, independent tasks at the same time.

Think of it like cooking a big meal. You don’t cook the chicken, then wait for it to finish before starting the potatoes, then wait again before making the salad. You put the chicken in the oven, put the potatoes on to boil, and chop the salad all at the same time. They all finish around the same time, and dinner is ready much faster.

In the previous article we covered the 5 main workflows when working with LLMs:

Also, previously we had covered the first, most basic one: “prompt chaining”:

In this issue we will:

• Define the “Parallelization” workflow pattern and explain its key benefits.

• Explore how to implement this pattern using LangChain.

• Build a step-by-step example that analyzes a news article in three different ways simultaneously. 💎 With all the code + notebook! 💎

Let’s begin!

Why is Parallelization so Effective?

• Speed and Efficiency: This is the biggest win. Instead of adding up the time for each task (Task A + Task B + Task C), the total time is only as long as your slowest task. This dramatically reduces latency, especially for complex queries.

• Rich, Structured Output: You can get a variety of different analyses from a single input, all neatly packaged into one structured object (like a dictionary or JSON).

• Task Isolation: Each “branch” of the parallel workflow runs independently. A failure or a poor-quality result in one branch (e.g., failing to find any “key phrases”) doesn’t stop the other branches (like “sentiment analysis”) from completing successfully.

This pattern is the perfect choice for any scenario where you need multiple different insights from the same single piece of input, and none of those insights depend on each other.

Under the Hood

LangChain Expression Language (LCEL) makes parallelization incredibly simple using a component called RunnableParallel.

More often, you’ll use its convenient shorthand: a Python dictionary.

When you define a step in your chain as a dictionary of other runnables, LCEL is smart enough to know it should execute all of them in parallel. The input to this dictionary step is passed to every runnable inside it, and the output is a dictionary where the keys are the same, but the values are the results of each runnable.

It looks like this:

parallel_step = {
    “output_key_1”: chain_1,
    “output_key_2”: chain_2,
}

If you invoke this with {”input”: “some data”}, LCEL will run chain_1 and chain_2 at the same time, both receiving that input. The final output will be {”output_key_1”: “result from chain 1”, “output_key_2”: “result from chain 2”}.

Let’s see this in action.

🛠️ Do It Yourself: Multi-Faceted Article Analysis

Imagine you have a stream of short news articles, and for your dashboard, you need to instantly extract three distinct pieces of information from each one.

The Challenge

We’ll take a single news blurb and run it through three parallel chains:

1. One-Sentence Summary: Generate a single, concise summary sentence.

2. Key Entity Extraction: Pull out all people, organizations, and locations into a structured JSON object.

3. Tone Analysis: Classify the article’s tone (e.g., Objective, Optimistic, Critical).

As an example, we will use the following news blurb:

“Quantum Leap Innovations (QLI), a tech startup based in Silicon Valley, announced yesterday it has secured $50 million in Series B funding. The round was led by Apex Ventures. CEO Dr. Aris Thorne stated the funds will be used to accelerate the development of their next-gen quantum computing platform, which aims to solve complex logistical problems.”

Let’s begin!

Setting the Stage

First, we need our LangChain tools and our LLM. We’ll also import Pydantic to define the exact structure we want for our extracted entities.

# Install necessary libraries if you haven’t already
# !pip install langchain-openai langchain pydantic

import os

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser, JsonOutputParser
from langchain_core.pydantic_v1 import BaseModel, Field

# Set up your OpenAI API key (replace with your actual key or environment variable)
# If your key is not already set as an environment variable, uncomment and run the following:
# os.environ[”OPENAI_API_KEY”] = "your_api_key"

# Initialize our AI brain (the LLM) using gpt-4o
llm = ChatOpenAI(model=”gpt-4o”)

Interpretation:

• ChatOpenAI(model=”gpt-4o”): This initializes our LLM.

• StrOutputParser: We’ll use this to get simple string outputs for our summary and tone.

• BaseModel, Field: These are from Pydantic. We’ll use them to create a “schema” or template for our entities.

• JsonOutputParser: This parser will take the LLM’s string output and automatically convert it into a Python dictionary that matches our Pydantic model.

📖 Book of the Week

If you are building, or want to build, real agentic systems in Python, this is a standout pick for your stack.

“Learn Model Context Protocol with Python” by Christoffer Noring

💡 A practical, end-to-end guide to Model Context Protocol (MCP) that shows you how to design, build, test, secure, and deploy interoperable AI applications using Python. You will move from first server and client to cloud deployment with tooling that works across LLM and non-LLM clients.

What sets it apart

It focuses on standardisation and interoperability with MCP, so your agents, tools, and hosts speak the same language. You get hands-on workflows for server and client development, clear security guidance, and real integrations with Claude Desktop and Visual Studio Code Agents.

You will learn to:

✅ Understand the MCP spec and core components
✅ Build MCP servers that expose tools and resources to many clients
✅ Describe host, client, and server capabilities for smooth interoperability
✅ Test and debug with interactive inspector tools
✅ Consume servers using Claude Desktop and VS Code Agents
✅ Secure MCP apps and mitigate common threats
✅ Deploy MCP apps using cloud-based strategies

Who should read this

🧠 Web developers, software architects, AI practitioners, and tech leads building scalable AI-integrated apps
📈 Product managers driving AI initiatives who need a shared language for teams
📚 Readers with basic web and AI knowledge who want production-ready patterns

Why it is useful for teams

• A single, modern approach for distributed agentic AI apps

• Professional guidance for both LLM and non-LLM clients

• Print or Kindle purchase includes a free PDF eBook, plus downloadable code

If you are serious about MCP and agentic architectures, this is the resource that turns experimentation into reliable AI systems.

Get it here

Now we are ready to build our parallel branches!

Branch 1: The Summary Chain

This is a simple chain to generate a one-sentence summary.

# Prompt for Summary
summary_prompt = ChatPromptTemplate.from_template(
    “Summarize the following article in one single, concise sentence.\n\nArticle:\n{article}”
)

# Define the summary sub-chain
summary_chain = summary_prompt | llm | StrOutputParser()

Interpretation:

• This is a standard sequential chain. It takes an article input, formats it with the prompt, sends it to the LLM, and parses the output as a string.

Branch 2: The Entity Extraction Chain

This branch is more advanced. We’ll define a Pydantic model for our desired output and use a JsonOutputParser to force the LLM’s response into that structure.

Read more

Traditional chatbots start fresh with every conversation—they have no memory of who you are or what you've discussed before. But what if your chatbot could remember past interactions and learn about users over time?

In this article, we'll build exactly that: a Python-based conversational agent using LangGraph that can remember user information across sessions and automatically update that information as conversations progress. This isn't just about storing data—it's about creating an agent that naturally extracts and organizes information from normal conversation flow.

We'll cover this in three parts:

1. The concept: What the agent does and why it's useful

2. A complete example: How the agent processes a real conversation

3. The implementation: How the code actually works (💎a colab notebook is also shared at the end with all the code💎)

Part 1: Understanding the Agent

Core Functionality

This agent operates on a dual-track system during every conversation:

1. Retrieves existing information: When you ask "What city do you have on file for me?" the agent looks up your stored profile and gives you an accurate answer. It doesn't guess or hallucinate—it checks the actual data and reports back what it finds.

2. Captures new information: When you mention "I just moved to London and started painting," the agent automatically:

• Recognizes these as new data points worth storing

• Extracts them into structured format (location: "London", hobbies: ["painting"])

• Saves them to your profile without interrupting the conversation

The magic is in the seamlessness. Users don't need to fill out forms or use special commands. They just talk naturally, and the agent handles both information retrieval and capture in the background. This creates a feedback loop where better memory leads to better conversations, which encourages users to share more, which improves the memory further.

Practical Applications

This architecture opens up several powerful use cases that go beyond simple Q&A:

• Better customer service: Imagine calling support and the bot already knows you own the Pro version, had a billing issue last month, and prefer email communications. It can skip the entire triage process and immediately offer relevant solutions. For businesses, this means faster resolution times and happier customers.

• Personal assistants that actually assist: A digital helper that remembers your daughter is allergic to peanuts, your anniversary is next week, and you're trying to learn Spanish can provide genuinely useful, proactive suggestions. It moves from being a reactive search tool to a proactive partner.

• Automated CRM updates: Sales teams spend hours logging client interactions. An agent that can parse "The client mentioned they're expanding to Europe next quarter and need enterprise pricing" and automatically update the CRM saves time while ensuring nothing gets forgotten.

• Educational companions: Tutoring bots that remember a student's weak areas, learning pace, and preferred explanation styles can provide truly personalized education that adapts over multiple sessions.

• Healthcare intake: Medical assistants that can naturally collect patient history during conversation, remembering symptoms mentioned across multiple visits and flagging important changes to doctors.

The ReAct Pattern

The agent uses a "Reason and Act" (ReAct) pattern—a cognitive framework that mimics how humans solve problems. Unlike simple if-then chatbots, ReAct agents can plan multi-step solutions and adapt their approach based on intermediate results.

Here's what happens when you send a message like "What are my hobbies? By the way, I'm 28 now and I also enjoy coding":

1. Receive: The agent gets your raw message exactly as typed

2. Reason: The LLM (Large Language Model) breaks down the message:

   • Identifies a question: "What are my hobbies?"

   • Spots new information: age update (28) and new hobby (coding)

   • Plans the sequence: fetch profile first, then extract new details

3. Act: It executes the plan by calling tools:

   • Calls get_user_profile(user_id=12345) to fetch current data

   • Calls extract_personal_details(age=28, hobbies=["coding"]) to structure the new info

4. Observe: The tools return their results:

   • Current profile shows existing hobbies: ["reading"]

   • Extraction confirms new data ready for storage

5. Respond: The agent:

   • Synthesizes a natural response: "According to your profile, your hobbies include reading. I'll also note that you enjoy coding and update your age to 28."

   • Triggers background profile update to merge new data

This pattern is powerful because the agent can adjust its approach mid-stream. If the profile lookup fails, it can acknowledge that and still save the new information. If the user's message is ambiguous, it can ask for clarification before proceeding.

📖 Book of the week

I just started reading The Definitive Guide to OpenSearch by Jon Handler, Soujanya Konka, and Prashant Agarwal — and it’s a must-have for anyone working with large datasets, search, or analytics.

What makes it stand out:

• ✅ Covers fundamentals and advanced optimizations

• ✅ Real-world case studies and hands-on demos

• ✅ Insights on scaling search and analytics systems

• ✅ Even explores Generative AI with OpenSearch

As data scientists and engineers, mastering search infrastructure is key to building scalable and intelligent systems — and this book brings best practices straight from AWS experts.

🔍 Highly recommended if you’re in AI/ML, data engineering, or system design.

👉 Check it out here: The Definitive Guide to OpenSearch

Part 2: A Complete Example

Let's walk through a real conversation to see what we aim to achieve.

Starting Scenario

Our user (ID: 12345) has this initial profile:

{
    "name": "John Doe",
    "age": 30,
    "city": "New York",
    "hobbies": ["reading"]
}

The Conversation

User says: "Hi, what are my hobbies? Btw, I just moved to San Francisco, I'm 28 now, and I also enjoy coding."

Let's trace what happens:

Step 1: Message Processing

The agent receives the message and begins reasoning. It identifies:

• A question that needs the current profile

• Three pieces of new information to extract

Step 2: Tool Execution

The agent makes two tool calls:

First tool call - Getting current profile:

--- TOOL: Looking up profile for user_id: 12345 ---
Returns: {"name": "John Doe", "age": 30, "city": "New York", "hobbies": ["reading"]}

Second tool call - Extracting new details:

--- TOOL: Extracting details: {
    'location': 'San Francisco', 
    'age': 28, 
    'hobbies': ['coding']
} ---

Step 3: Response Generation

The agent crafts a natural response:

"Based on your profile, your current hobbies include reading. I've noted that you also enjoy coding now. I've also updated your location to San Francisco and your age to 28. Is there anything specific you'd like to know about your updated profile?"

Step 4: Profile Update

Behind the scenes, the smart update function processes the new details:

# Before update:
{
    "name": "John Doe",
    "age": 30,
    "city": "New York", 
    "hobbies": ["reading"]
}

# After smart merge:
{
    "name": "John Doe",
    "age": 28,                    # Updated scalar
    "city": "New York",           # Unchanged
    "location": "San Francisco",  # New field added
    "hobbies": ["reading", "coding"]  # List extended
}

Notice how:

• The age was updated from 30 to 28

• "location" was added as a new field (the agent used "location" instead of "city" for San Francisco)

• The hobbies list was extended, not replaced

• The name remained unchanged

Following Up

If the user continues the conversation:

User: "Actually, I also enjoy hiking and photography."

The agent would:

1. Extract hobbies: ["hiking", "photography"]

2. Merge them into the existing list

3. End up with hobbies: ["reading", "coding", "hiking", "photography"]

This demonstrates the power of the append logic—information accumulates naturally over multiple interactions.

Part 3: Implementation Details

Now that we know what we want to achieve, let's examine how to do it, piece by piece.

1. State Management

The agent's working memory is deliberately minimal—just enough to maintain context without unnecessary complexity:

class MyState(AgentState):
    user_id: int

• user_id: The unique identifier that links this conversation to a specific user's data

• messages: Inherited from AgentState, this list tracks the entire conversation history including human inputs, AI responses, tool calls, and tool results

Why so simple? The state is transient—it only exists for the duration of a conversation. The permanent user data lives elsewhere (in a database or file), and the state just needs enough information to access it. This separation keeps the agent lightweight and allows the storage backend to be swapped out without changing the agent logic.

The messages list is particularly important because it provides the full audit trail. Every decision the agent makes, every tool it calls, and every result it receives gets logged here. This is invaluable for debugging and understanding the agent's reasoning process.

Read more

Imagine you have a complex task, like writing a detailed report or analyzing a lengthy customer feedback document. Trying to get an AI to do it all in one go can be like asking a chef to prepare a gourmet meal with just one instruction: "Make dinner!" The result might be edible, but probably not perfect.

This is where Prompt Chaining comes in. It's a powerful, yet elegantly simple, workflow pattern that helps Large Language Models (LLMs) tackle intricate problems by breaking them down into a series of smaller, more manageable steps.

Think of it like cleaning your house room by room instead of trying to tackle the whole place at once. When you focus on just the kitchen, you actually get it spotless. Try to clean everything simultaneously, and you'll probably just move clutter around.

Why is Prompt Chaining so Effective?

• Enhanced Accuracy: By focusing on one sub-task at a time, the LLM can dedicate its full attention and reasoning power to that specific step, leading to more precise and reliable outputs for each stage.

• Clarity and Control: You gain a clearer understanding of how the AI arrives at its final answer, making debugging and refinement much easier. Each step is transparent.

• Modularity: Each part of your chain is a distinct component. This means you can easily swap out or improve individual steps without disrupting the entire workflow.

This pattern is particularly well-suited for tasks that naturally have a clear, sequential flow, where the successful completion of one stage is a prerequisite for the next. For example, generating a document outline, then checking that outline against specific criteria, and finally writing the document based on the refined outline.

Under the Hood

LangChain Expression Language (LCEL) makes building these chains incredibly intuitive. LCEL uses Python's familiar pipe operator (|) to connect different components, creating a RunnableSequence. This means the output of one component automatically flows as the input to the next, just like water through a pipe!

Every core component in LangChain – from your prompts to your LLMs and output parsers – implements the Runnable protocol, making them perfectly compatible with this chaining mechanism.

Let's see how this works in practice with a real-world example.

🛠️Do It Yourself: Refining Sentiment Analysis

Imagine you have a stream of customer feedback, and you need to not only understand the sentiment but also extract key issues and present a concise, refined summary. This is a perfect candidate for prompt chaining.

The Challenge

We'll take a customer review and put it through a multi-stage process:

1. Sentiment Analysis: Determine the overall sentiment (Positive, Negative, or Neutral) and provide a brief explanation.

2. Key Phrase Extraction: Identify up to 5 important key phrases that capture the essence of the review.

3. Refined Summary: Create a concise 2-3 sentence summary incorporating the original review, its sentiment, and the extracted key phrases.

As an example we will use the following customer review:

“The new coffee machine is a disaster. It constantly leaks water, the coffee tastes burnt, and the brewing process takes forever. I'm extremely disappointed with this purchase and would not recommend it to anyone. The previous model was much better.”

Let’s begin!

Setting the Stage

First, we need to import the necessary tools from LangChain and initialize our Large Language Model. We'll use ChatOpenAI for this example, specifically the gpt-4o model.

# Install necessary libraries if you haven't already
# !pip install langchain-openai

import os
from getpass import getpass

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from operator import itemgetter
from langchain_core.output_parsers import StrOutputParser

# Set up your OpenAI API key (replace with your actual key or environment variable)
# If your key is not already set as an environment variable, uncomment and run the following:
# os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API Key:")

# Initialize our AI brain (the LLM) using gpt-4o
llm = ChatOpenAI(model="gpt-4o")

Interpretation:

• !pip install langchain-openai: This command installs the required LangChain and OpenAI libraries in your environment.

• os.environ["OPENAI_API_KEY"] = getpass(...): This line is for securely setting your OpenAI API key as an environment variable, which is crucial for authenticating with OpenAI's models.

• ChatOpenAI(model="gpt-4o"): This initializes our LLM instance, specifically using the gpt-4o model.

• ChatPromptTemplate, RunnablePassthrough, RunnableLambda, itemgetter, StrOutputParser: These are the core LangChain components we'll use to build our flexible and powerful chains.

Understanding Pure Sequential Chaining

Before diving into our multi-faceted example, it's helpful to see a simpler form of chaining where the output of one step directly becomes the sole input of the next. This illustrates the fundamental sequential flow.

# A simple, pure sequential chain example
pure_sequential_example = (
    ChatPromptTemplate.from_template("What is the capital of {country}?")
    | llm
    | StrOutputParser()
    | ChatPromptTemplate.from_template("Briefly tell me about {text}.") # {text} here would be the capital from the previous step
   | llm
   | StrOutputParser()
)

# Invoke the simple sequential chain
print(pure_sequential_example.invoke({"country": 'Spain'}))

The output of this code would be:

Madrid is the capital and largest city of Spain. It is located in the center of the country and serves as its political, economic, and cultural hub. Known for its vibrant nightlife, historic architecture, world-class museums like the Prado and Reina Sofia, and lively neighborhoods, Madrid is a city that seamlessly blends traditional charm with modern energy. It is also home to the Spanish royal family and government institutions.

Interpretation:

This simple chain first asks the LLM for the capital of a given country, then takes that capital as the input ({text}) for a second prompt, asking for information about it. This perfectly illustrates how the output of one Runnable (the capital city) becomes the input for the next Runnable (ChatPromptTemplate.from_template("Briefly tell me about {text}.")).

📖 Book of the Week

If you're building or want to build LLM-powered apps and agents — whether you're an AI developer, MLOps engineer, or product team lead — you need to check this out:

“Generative AI with LangChain” (Second Edition)
By Ben Auffarth & Leonid Kuligin

💡 This book isn’t just about building cool prototypes — it’s a practical guide to designing, scaling, and deploying production-ready GenAI systems using LangChain, LangGraph, and Python.

What sets it apart?

It tackles one of the biggest challenges in GenAI: moving from prototype to production — with a strong focus on multi-agent coordination, observability, and real-world deployment:

• ✅ Design robust LangGraph agent workflows that scale

• ✅ Build powerful RAG pipelines with re-ranking and hybrid search

• ✅ Apply enterprise-ready testing, monitoring, and error-handling

• ✅ Explore Tree-of-Thoughts, structured generation, and agent handoffs

• ✅ Work with top LLMs like Gemini, o3-mini, Mistral, DeepSeek, and Claude

This is the guide that turns experimentation into reliable AI infrastructure.

This is a must-read for:

• 🧠 AI engineers building multi-agent systems

• 🐍 Python devs deploying LLM apps in real-world environments

• 🏢 Enterprise teams moving GenAI projects into production

• 🔬 Anyone working with LangChain, LangGraph, or advanced RAG

• ✅ Devs who care about security, compliance, and ethical AI

Get it here

Now we are ready for the more complex scenario!

Sentiment Analysis

Our first task is to determine the sentiment of the customer review. We'll define a prompt specifically for this and create a simple chain.

# Prompt for Sentiment Analysis
sentiment_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert sentiment analyzer. Analyze the sentiment of the following customer review and categorize it as 'Positive', 'Negative', or 'Neutral'. Provide a brief explanation for your categorization."),
        ("user", "Customer Review:\n{review}"),
    ]
)

# Define the sentiment analysis sub-chain
sentiment_analysis_chain = sentiment_prompt | llm | StrOutputParser()

Interpretation:

• sentiment_prompt: This ChatPromptTemplate is designed to take a review as input. The system message instructs the AI to act as a sentiment analyzer and categorize the sentiment.

• sentiment_analysis_chain = sentiment_prompt | llm | StrOutputParser(): This creates our first simple chain. It pipes the review into the sentiment_prompt, sends the result to our llm (gpt-4o), and then uses StrOutputParser() to convert the LLM's output into a clean string.

We can invoke this chain with the customer review and see what is the output:

print(sentiment_analysis_chain.invoke({"review": customer_review}))

Sentiment: Negative

Explanation: The review expresses clear dissatisfaction with the product, describing it as a "disaster" and highlighting several specific issues, including water leaks, burnt coffee flavor, and a long brewing process. Additionally, the reviewer explicitly states being "extremely disappointed" and advises against purchasing the machine, further reinforcing the negative sentiment.

Key Phrase Extraction

Next, we'll focus on extracting key phrases from the original review. This will be another independent step, so it gets its own prompt and simple chain.

# Prompt for Key Phrase Extraction
key_phrases_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert in extracting key phrases. From the following customer review, identify and list up to 5 of the most important key phrases that capture the essence of the review. Present them as a comma-separated list."),
        ("user", "Customer Review:\n{review}"),
    ]
)

# Define the key phrase extraction sub-chain
key_phrase_extraction_chain = key_phrases_prompt | llm | StrOutputParser()

Interpretation:

• key_phrases_prompt: This prompt focuses solely on extracting important key phrases from the review.

• key_phrase_extraction_chain = key_phrases_prompt | llm | StrOutputParser(): Similar to the sentiment chain, this forms a self-contained unit that takes a review and returns a comma-separated list of key phrases.

Similarly, we can invoke this chain and see its output:

print(key_phrase_extraction_chain.invoke({"review": customer_review}))

leaks water, coffee tastes burnt, brewing process takes forever, extremely disappointed, previous model was better

Refined Summary

Finally, we define the prompt for generating the refined summary. This prompt is special because it will require inputs from the original review and the outputs of our previous two sub-tasks.

# Prompt for Refined Summary using outputs from previous steps
summary_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert summarizer. Based on the following customer review, its sentiment, and key phrases, create a concise, 2-3 sentence summary that incorporates these details."),
        ("user", "Customer Review:\n{review}\n\nSentiment:\n{sentiment_analysis}\n\nKey Phrases:\n{key_phrases}"),
    ]
)

Interpretation:

• summary_prompt: This prompt expects three distinct inputs: review, sentiment_analysis, and key_phrases. The system message guides the AI to combine these elements into a concise summary. Notice that we don't define a full chain for this yet, as it needs to receive inputs from the previous dynamic steps.

Building the Full Chain

Read more

Large Language Model (LLM) agents can make decisions about when to use external tools as part of answering a question.

We covered AI Agents in this previous issue:

Let’s now cover the most basic agent: ReAct agent. The ReAct (Reasoning and Acting) style agent operates in a loop of:

1. thinking (reasoning with the LLM)

2. acting (calling a tool or API)

3. observing (incorporating the tool's result)​

This allows the agent to handle queries that the LLM alone might not answer, by dynamically invoking tools for additional information​. For example, if asked “What’s the GDP of Spain in 2024?”, a ReAct agent could decide to call a Wikipedia search tool to fetch the latest data.

In this issue, we will build a simple ReAct-style agent from scratch using LangGraph (LangChain's graph-based framework) and LangChain in Python.

We will not use any pre-built agent utilities; instead, we'll explicitly define the agent's graph nodes and conditional edges. The agent will be able to use a Wikipedia search tool automatically when needed.

In this issue, we will cover a Stateless Agent (single-turn), which is a minimal ReAct agent that answers one question at a time without conversation memory. However, in future articles we will also cover a Stateful Agent (with memory) - an extension that keeps track of the conversation history so it can handle follow-up questions.

Let's get started by setting up our environment and then implementing the agent step by step.

Setup

First, install the required packages and set up any API keys. We'll use LangGraph (part of LangChain for building graph-based LLM workflows), LangChain's OpenAI chat model wrapper, and the wikipedia package for the Wikipedia search tool.

!pip install -U langgraph langchain-openai wikipedia

This imports the necessary classes and functions:

• LangChain’s chat model (we'll use OpenAI's GPT-4o for demonstration via ChatOpenAI),

• LangChain’s tool decorator (@tool) to define our custom tool,

• LangGraph’s components for building the state graph,

• Message classes for constructing the conversation state.

from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.graph import StateGraph, END
from langchain_core.messages import BaseMessage, SystemMessage, ToolMessage

Note: You will need an OpenAI API key if using ChatOpenAI. Make sure to set OPENAI_API_KEY as an environment variable or via os.environ before running the agent, or directly passing through the api_key argument. Alternatively, you could use a local or open-source model with a similar interface.

Now, let's implement the agent!

Agent

In this section, we build a minimal ReAct agent that can answer one question (with tool use if needed) and does not retain any memory of previous interactions. The agent will use the LLM to decide on actions and will handle the reasoning-action loop for a single query.

We will implement the agent step by step:

1. Define the agent's state – the data structure representing the agent's memory or context (for a stateless single-turn agent, this will just include the current conversation messages).

2. Set up the LLM and tool – initialize the language model and define a Wikipedia search tool using the wikipedia package. Give tools access to the model.

3. Define LangGraph nodes and edges – create the reasoning node (LLM call), the tool node (executes the tool), and a conditional edge that decides whether to continue the loop or end it, based on the LLM's output.

4. Compile and run the graph – combine the nodes into a StateGraph, then test the agent on a sample question to see it in action.

1. Defining the State Model

LangGraph uses a state object to keep track of the conversation and any intermediate data. For a basic ReAct agent, the state can be as simple as a list of messages (chat history). We define a TypedDict for the state with a single key "messages" that will hold a sequence of messages. We also attach a reducer add_messages to this field – this ensures that when we return new messages from a node, they get appended to the state’s message list (instead of overwriting it)​.

from typing import TypedDict, Sequence, Annotated
from langgraph.graph.message import add_messages

class AgentState(TypedDict):
    """State of the agent for one turn or conversation."""
    messages: Annotated[Sequence[BaseMessage], add_messages]

Here, AgentState["messages"] will contain a sequence of chat messages (from system, user, AI, or tool). For a single-turn stateless agent, we'll initialize this with just the latest user question (and a system prompt internally).

2. Setting up the LLM and Wikipedia Tool

Next, initialize the LLM and define our tool. We use ChatOpenAI from LangChain to create a chat model instance. For demonstration, we'll use the "gpt-4o" model (which supports OpenAI's function calling), but you can use any model available (as long as it is compatible with tools calling).

We then define a Wikipedia search tool using the @tool decorator. This decorator turns a Python function into a LangChain tool that the agent can call. Our tool function will take a search query string, use the wikipedia library to fetch a summary of the top result, and return that summary text. The docstring of the function serves as the tool’s description for the LLM.

# Initialize the chat model (LLM) - make sure your API key is set
model = ChatOpenAI(model="gpt-4o", temperature=0, api_key=OPENAI_API_KEY)

# Define a Wikipedia search tool
import wikipedia

@tool
def wiki_search(query: str) -> str:
    """Search Wikipedia for the query and return a brief summary of the top result."""
    try:
        # Fetch summary of top search result (we set it to 5 sentences)
        summary = wikipedia.summary(query, sentences=5)
        return summary
    except Exception as e:
        return f"Error: {e}"

We set a low temperature for the model to minimize randomness, since we want it to reliably produce tool calls for unknown facts. If the API key was not set in the environment you also need to pass it as an argument.

The wiki_search tool uses the Wikipedia API (via the wikipedia package) to get information. For example, if asked about a person or event not known to the model, the agent can call wiki_search to get up-to-date info​. This is a common pattern – Wikipedia tools are often used to fetch summaries for factual questions​. For this example we set the number of sentences of the summary to 5, but that is something that can be changed according to your application needs.

3. Defining the LangGraph Nodes and Conditional Logic

Before defining the graph, let’s prepare the tools so the model can access them:

import json

# Map tool name to the tool function for easy lookup
tools = [wiki_search]
tools_by_name = {tool.name: tool for tool in tools}

# Give the model access to the tools
model = model.bind_tools(tools) 

With the model and tool ready, we create the nodes of our agent's computation graph:

• Reasoner Node (LLM call): This node will call the LLM to either produce an answer or decide on a tool action. We’ll implement it as a function call_model(state). It takes the current state (which contains the conversation messages so far) and returns the LLM's response as a new message. We include a system prompt to guide the LLM’s behavior (e.g., “You are a helpful assistant…”). The user’s query is in the state’s messages. We invoke the model with the system prompt plus all existing messages. LangChain’s ChatOpenAI can return a message that includes a function call if the model decides a tool is needed (under the hood, the model may use OpenAI’s function calling feature to request wiki_search).

def call_model(state: AgentState):
    """LLM reasoning node: call the chat model with system prompt + conversation."""
    system_prompt = SystemMessage(content="You are a helpful AI assistant. If needed, you can use the wiki_search tool to build your answer.")
    # Call the chat model with system + existing messages (user question is included in state["messages"])
    response = model.invoke([system_prompt] + list(state["messages"]))
    # Return the response as a list (to be appended to state's messages via reducer)
    return {"messages": [response]}

• Tool Node (execute tool): This node executes any tool that the LLM requested. We implement tool_node(state) to inspect the latest message from the LLM for a tool call. If a tool call is present, we invoke the corresponding tool function and package its result into a special ToolMessage. The ToolMessage will be added to the state so the LLM can see the tool’s output on the next iteration.

def tool_node(state: AgentState):
    """Tool execution node: execute any tool calls the LLM asked for."""
    outputs = []
    # Check the last message from the LLM for tool calls
    last_message = state["messages"][-1]
    if hasattr(last_message, "tool_calls") and last_message.tool_calls:
        # If the model requested one or more tool calls, execute each
        for tool_call in last_message.tool_calls:
            tool_name = tool_call["name"]
            tool_args = tool_call["args"]
            if tool_name in tools_by_name:
                # Invoke the corresponding tool function with provided arguments
                result = tools_by_name[tool_name].invoke(tool_args)
            else:
                result = f"Tool '{tool_name}' not found."
            # Wrap the result in a ToolMessage for the LLM to read
            outputs.append(
                ToolMessage(
                    content=json.dumps(result),   # tool result as JSON string
                    name=tool_name, 
                    tool_call_id=tool_call.get("id")  # use id if provided
                )
            )
    # Return the tool outputs to be added to messages
    return {"messages": outputs}

• Conditional Edge (should_continue): After each LLM reasoning step, we need to decide whether the agent should end with an answer or continue by using a tool. We define a function should_continue(state) that checks the LLM's last message. If the LLM did not request any tool (no function call), that means it produced a final answer, so the agent can end. If a tool was requested, we should continue to the tool node next. This function will return a flag (e.g., "continue" or "end") that LangGraph uses to choose the next node.

def should_continue(state: AgentState) -> str:
    """Decide whether to continue the ReAct loop or end it, based on last LLM message."""
    last_message = state["messages"][-1]
    print(last_message)
    # If the LLM's last message did not request a tool, we're done
    if not (hasattr(last_message, "tool_calls") and last_message.tool_calls):
        return "end"
    else:
        # There is a tool request, so continue to the tool node
        return "continue"

A few notes on this implementation:

• In call_model, we prepend a system message that defines the assistant's role and hints that it can use the wiki_search tool if needed. We then pass all messages (including the user's message) to the chat model. The model may return a normal AI message (with a direct answer) or a function/tool call message. LangChain's ChatOpenAI will automatically format the function call request in a structured way if the model decides to use a tool.

• In tool_node, we look at last_message.tool_calls. LangChain’s message objects have a tool_calls attribute that contains any tool/function call requests the model made​. If there's a tool call, it includes the tool name and args. We invoke the appropriate tool from our tools_by_name registry. The result is wrapped in a ToolMessage which includes the tool’s name and outputs. By returning {"messages": [ToolMessage(...)]), LangGraph's reducer will append this tool result message to the state’s messages list.

• should_continue examines the last message. If tool_calls is empty, the LLM didn't ask for any action – meaning it likely produced a final answer – so we return "end". If there's a tool call, we return "continue", signaling the graph to proceed to the tool execution step​. These return strings will be used to choose the next node via a conditional mapping.

‍🎓Further Learning*

Let us present: “From Beginner to Advanced LLM Developer”. This comprehensive course takes you from foundational skills to mastering scalable LLM products through hands-on projects, fine-tuning, RAG, and agent development. Whether you're building a standout portfolio, launching a startup idea, or enhancing enterprise solutions, this program equips you to lead the LLM revolution and thrive in a fast-growing, in-demand field.

Who Is This Course For?

This certification is for software developers, machine learning engineers, data scientists or computer science and AI students to rapidly convert to an LLM Developer role and start building

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

4. Constructing and Compiling the Graph

Now we assemble the graph using LangGraph’s StateGraph. We add our two nodes ("agent" for the LLM reasoning and "tool" for the tool execution), set the entry point, and define the transitions. The critical part is adding a conditional edge from the LLM node to either the tool node or the end of the graph, based on should_continue function's output. We will map the "continue" signal to the "tool" node, and the "end" signal to END (a special marker indicating the graph should terminate). We also add a normal edge from the tool node back to the LLM node, creating a cycle: after using the tool, the agent goes back to the LLM to incorporate the new information.

Read more

Today we’re diving into sentiment analysis—a cool way to teach our models to decide if a movie review is cheering us on or giving us the boot. We’ll first introduce the NLP technique “Bag-of-Words” and then we’ll use an IMDb movie reviews dataset, clean the text, tokenize it, create a bag-of-words representation, and finally train a logistic regression model. Let’s get our hands dirty and see exactly what’s going on!

Thanks to for all the code and explanations!

Bag-of-Words

Before we jump into the coding fun, let's explore the Bag-of-Words (BoW) method, which serves as a fundamental building block in natural language processing and text analysis.

What It Is

Bag-of-Words is a simple yet powerful technique that converts text documents into numerical vectors that machine learning algorithms can understand. The name "bag" comes from the fact that this method disregards grammar and word order, treating text as an unordered collection (or bag) of words.

The method works by:

1. Splitting each text (or review) into individual words (tokens).

2. Building a vocabulary of all unique words found across all documents in the corpus.

3. Representing each document as a numerical vector where each element corresponds to the count of a specific word from the vocabulary.

The Historical Context

The Bag-of-Words approach has roots in information retrieval systems from the 1950s and gained popularity in the 1990s with the growth of digital document collections. Despite its simplicity compared to modern deep learning approaches, BoW remains relevant because it's computationally efficient and surprisingly effective for many text classification tasks, including sentiment analysis.

Why It Works

Since natural language contains thousands or even millions of unique words, but each individual document (like a movie review) only uses a small subset of those words, most positions in these vectors are zero. This "sparsity" makes it computationally efficient to store and process, even for large datasets.

Additionally, certain words strongly correlate with sentiment. For example, words like "excellent" and "terrible" are powerful indicators of positive and negative sentiment, respectively.

The BoW model captures these correlations effectively, making it well-suited for sentiment analysis tasks.

A Concrete Example

Let's bring this to life with a more detailed example to really understand how BoW works:

Imagine we have three movie reviews in our corpus:

1. "The movie was excellent and I enjoyed it."

2. "The acting was terrible but the plot was good."

3. "I thought the film was boring and predictable."

The BoW approach transforms these text documents into numerical vectors through a systematic process:

📌 Key Insights:

1. BoW ignores word order and just counts occurrences

2. Documents with similar sentiment often share key words

3. Notice how sentiment words like "excellent", "terrible", and "boring" stand out in their respective documents

4. Perfect for sentiment analysis as positive/negative words get their own dimensions

5. Each review becomes a point in a multi-dimensional space where ML algorithms can find patterns

Limitations and Considerations

While powerful, BoW has some limitations to keep in mind:

• It ignores word order, which can sometimes be important ("not good" has a different meaning than "good not").

• It creates high-dimensional, sparse vectors when vocabulary is large.

• Common words like "the" or "and" may dominate counts without contributing much meaning.

We'll address some of these limitations in our preprocessing steps, and many advanced NLP techniques build upon this foundation by addressing these weaknesses.

Now that we understand the conceptual framework, let's dive into implementing Bag-of-Words for our sentiment analysis task!

‍🎓Further Learning*

Let us present: “From Beginner to Advanced LLM Developer”. This comprehensive course takes you from foundational skills to mastering scalable LLM products through hands-on projects, fine-tuning, RAG, and agent development. Whether you're building a standout portfolio, launching a startup idea, or enhancing enterprise solutions, this program equips you to lead the LLM revolution and thrive in a fast-growing, in-demand field.

Who Is This Course For?

This certification is for software developers, machine learning engineers, data scientists or computer science and AI students to rapidly convert to an LLM Developer role and start building

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

🛠️ Do It Yourself

Since last week we missed the issue, this one we will compensate with a longer one: theory + practice! Apologies for that and I hope you like it!

At the end, as usual, you’ll get a notebook with all the code!!

These are the steps:

1. Load and Prepare Data: Import necessary tools and load the movie review data, setting the stage for analysis.

2. Clean the Text: Remove unwanted elements like HTML and punctuation, making the text ready for analysis by standardizing it.

3. Convert Text to Numbers: Transform the cleaned text into numerical data using a "bag-of-words" model, so the computer can understand it.

4. Train a Predictive Model: Split the data into training and testing sets, and train a logistic regression model to predict sentiment.

5. Analyze the Results: Interpret the model's performance and understand which words contribute to positive or negative reviews.

Let’s finally begin!

Step 1: Importing Libraries & Loading the Data

First things first: we import necessary libraries and load the data from GitHub.
We’re using pandas to load a CSV file directly from a GitHub repository that contains thousands of IMDb reviews. This dataset has two columns: one for the review text and one for the sentiment (labeled "positive" or "negative").

import numpy as np
import pandas as pd

# Set a fixed seed to ensure our results are reproducible.
np.random.seed(42)

# Load the IMDb movie reviews dataset from GitHub.
# Note: This dataset contains text reviews and sentiment labels.
url = (
    "https://raw.githubusercontent.com/Ankit152/IMDB-sentiment-analysis/refs/heads/master/IMDB-Dataset.csv"
)
df = pd.read_csv(url)

# Check the shape of the data (rows, columns) and preview the first few records.
print("Data shape:", df.shape)
print(df.head())

What’s happening?

• We set the seed for reproducibility.

• We load the CSV from GitHub.

• We print out the shape and a preview so we know the data is what we expect.

Step 2: Preprocessing the Text

Before we train a model, our text needs some cleaning. This step removes unwanted HTML tags, punctuation, and normalizes everything to lowercase. We also extract emoticons because they can be full of sentiment (think of :-) versus :().

import re

def preprocessor(text):
    """
    Clean the input text:
    - Remove HTML markup.
    - Extract emoticons and preserve them.
    - Remove non-word characters (like punctuation) and convert to lowercase.
    - Append cleaned emoticons (without the hyphen) back to the text.
    """
    # Remove HTML tags using regex
    text = re.sub(r"<[^>]*>", "", text)
    
    # Find emoticons (patterns like :), :-), :D, etc.)
    emoticons = re.findall(r"(?::|;|=)(?:-)?(?:\)|\(|D|P)", text)
    
    # Remove non-word characters, change text to lowercase, and append emoticons at the end.
    text = re.sub(r"[\W]+", " ", text.lower()) + " " + " ".join(emoticons).replace("-", "")
    return text

# Apply the preprocessor to our reviews
df["review_clean"] = df["review"].apply(preprocessor)

# Print a sample cleaned review (displaying the last 100 characters for brevity)
print("\nSample cleaned review:", df.loc[0, "review_clean"][-100:])

More details:

• Removing HTML Tags (<[^>]*>)
  • Looks for the < character and captures everything until the next >, removing HTML elements such as <div> or <br>.

• Extracting Emoticons
  • The pattern (?::|;|=)(?:-)?(?:\)|\(|D|P) finds common emoticons (like :-), :D), ensuring emotional cues are not lost.

• Cleaning Text with [\W]+
  • Replaces sequences of non-word characters (anything besides letters, digits, or underscores) with a space, standardizing the text while keeping the emoticons to preserve sentiment.

Step 3: Tokenizing & Building the Bag-of-Words Model

Tokenization is simply breaking text into individual words (tokens). The bag-of-words approach will create a vocabulary of unique words and count the number of times each appears in a review. Here, we use scikit-learn’s CountVectorizer to do the heavy lifting.

from sklearn.feature_extraction.text import CountVectorizer

# Instantiate the CountVectorizer.
# This will first split our text into tokens and then count occurrences.
vectorizer = CountVectorizer()

# Fit the vectorizer on our cleaned review texts and transform them into numerical feature vectors.
X = vectorizer.fit_transform(df["review_clean"])

# Let’s inspect a small portion of the resulting vocabulary.
print("\nSample vocabulary mapping (word -> index):")
sample_vocab = dict(list(vectorizer.vocabulary_.items())[:10])
print(sample_vocab)

# Also print the bag-of-words array for the first 3 reviews.
print("\nBag-of-words representation for the first 3 reviews:")
print(X[:3].toarray())

Explanation:

• Tokenization & Vocabulary Building:
  • CountVectorizer splits each review into words (tokens) and collects unique words into a large vocabulary.

• Vector Representation:
  • Each review is then transformed into a vector where each number shows how many times a word from the vocabulary appears.

• Sparse Matrix:
  • Since most reviews only use a small fraction of the full vocabulary, many entries are zero, resulting in a sparse matrix.

• Example:

  

Step 4: Splitting Data & Training a Logistic Regression Model

Next, we transform the sentiment labels into a binary format (e.g., positive = 1, negative = 0), split our data into training and test sets, and then train a logistic regression classifier. Logistic regression is a popular choice due to its simplicity and interpretability—its coefficients tell us which words influence predictions.

Read more

Have you ever trained a machine learning model and thought, “Okay, it works… but why does it work?” Or maybe you’ve been asked, “Why did the model make this prediction?” and found yourself fumbling for an answer.

Well, you’re not alone. Machine learning models can feel like mysterious black boxes sometimes. But today, we’re going to crack that box open with SHAP (SHapley Additive exPlanations). SHAP is like a detective for your model—it tells you exactly how each feature contributes to a prediction.

Do you want more info? You can check our previous article:

🛠️ DIY

In this post, we’ll:

1. Load the California Housing dataset.

2. Train a simple linear regression model.

3. Use SHAP to explain the model’s predictions.

4. Create four awesome visualizations to understand feature importance and individual predictions.

By the end, you’ll not only understand your model better but also have some cool visualizations to show off.

✏️ Article and code by .

Let’s dive in!

Step 1: Importing Libraries

Before we start, let’s grab the tools we need. Think of this as packing your backpack before a hike.

# --- NUMERICAL COMPUTATION ---
import numpy as np  # Provides fast array operations and random number functions

# --- DATA MANIPULATION ---
import pandas as pd  # Allows us to work with tabular (DataFrame) data

# --- PLOTTING LIBRARIES ---
import matplotlib.pyplot as plt  # Essential for creating plots
import seaborn as sns  # Enhances the aesthetics of our matplotlib plots

# --- MODELING LIBRARY ---
from sklearn.linear_model import LinearRegression  # Implements simple linear models

# --- MODEL EXPLANATION ---
import shap  # Enables us to explain machine learning predictions using SHAP values

# Set a fixed seed to ensure our results are reproducible every time we run the script.
np.random.seed(42)

Why these libraries?

• NumPy & Pandas: To handle and manipulate data.

• Matplotlib & Seaborn: To create beautiful visualizations.

• scikit-learn: To train a simple, interpretable linear regression model.

• SHAP: The star of the show—used to explain model predictions.

Step 2: Loading the Data

Let’s load the California Housing dataset. This dataset contains information about housing blocks in California, such as median income, house age, and population.

# Load the California Housing dataset
X, y = shap.datasets.california(n_points=1000)

# Create a background dataset for SHAP analysis
# This background dataset is used to estimate the expected values when features are "missing" in SHAP value calculation
X_background = shap.utils.sample(X, 100)

What’s happening here?

• shap.datasets.california: Loads the dataset.

• X: The features (e.g., median income, house age).

• y: The target variable (median house value).

• X_background: A smaller sample of the data used by SHAP to calculate expected predictions.

Step 3: Training the Model

Now that we have the data, let’s train a simple linear regression model.

# Initialize the Linear Regression model
model = LinearRegression()

# Fit the model to the data
model.fit(X, y)

Why linear regression?

• It’s simple and interpretable.

• Each feature gets a coefficient, which tells us how much it contributes to the prediction.

At the moment we are this step (image taken from previous article), we have our kind of black box model, that we is working well but we don’t know exactly why:

Step 4: Analyzing Model Coefficients

Here’s a question: Which features matter the most in predicting house prices?

Let’s look at the model’s coefficients to find out:

print("Model Coefficients:")
for idx, feature in enumerate(X.columns):
    coef = model.coef_[idx]
    direction = "increases" if coef > 0 else "decreases"
    print(f"{feature}: {coef:.5f} ({direction} house value)")

Model Coefficients:
MedInc: 0.42563 (increases house value)
HouseAge: 0.01033 (increases house value)
AveRooms: -0.11610 (decreases house value)
AveBedrms: 0.66385 (increases house value)
Population: 0.00003 (increases house value)
AveOccup: -0.26096 (decreases house value)
Latitude: -0.46734 (decreases house value)
Longitude: -0.46272 (decreases house value)

What does this tell us?

• A positive coefficient means the feature increases house value.

• A negative coefficient means the feature decreases house value.

For example, if the coefficient for MedInc (median income) is 0.5, it means that for every 1-unit increase in MedInc, the predicted house value increases by 0.5 units.

Step 5: Explaining Predictions with SHAP

Here’s where the magic happens. Let’s use SHAP to explain our model’s predictions. Something like this (from our previous article):

Initializing the SHAP Explainer

# Create a SHAP KernelExplainer using our model’s predict function, with the background dataset.
explainer = shap.KernelExplainer(model.predict, X_background)

What’s happening here?

• KernelExplainer: Works with any model (even non-linear ones).

• Background dataset: Helps SHAP calculate the expected prediction when features are “missing.”

Computing SHAP Values

# Compute SHAP values for the first 100 samples
X_display = X.iloc[:100]
shap_values = explainer.shap_values(X_display)

Why only 100 samples?
SHAP calculations can be computationally expensive. Using a subset makes it faster while still providing meaningful insights.

Before you continue… Have you ever wondered how to build an AI agent that thinks and acts intelligently? Creating one requires several key steps. Check out this article from our colleagues at Everyday | AI for all to learn more about this:

Everyday | AI for all

#40 The guide for developing AI Agents succesfully

Have you ever wondered how to create an AI agent that thinks and acts intelligently…

Read more

a year ago · 3 likes · Everyday by Pol and David Andrés

Step 6: Creating SHAP Visualizations

Let’s create some awesome visualizations to understand our model better.

SHAP Summary Plot

Question: Which features have the biggest impact on predictions?

sns.set_style("whitegrid")
plt.figure(figsize=(14, 10))
shap.summary_plot(shap_values, X_display, show=False)
plt.title("SHAP Summary Plot: Distribution of Feature Impacts", fontsize=14)
plt.xlabel("SHAP Value (Effect on Prediction)", fontsize=12)
plt.ylabel("Features", fontsize=12)
plt.tight_layout()
plt.show()

What does this plot show?
This plot is like a feature leaderboard. The most important features are at the top, and the least important ones are at the bottom. Each dot represents a single data point, and the position of the dot shows how much that feature influenced the prediction.

• Red dots: High feature values.

• Blue dots: Low feature values.

For example, if you look at MedInc (median income), you’ll see that higher values (red dots) tend to push predictions up, while lower values (blue dots) pull them down.

Partial Dependence Plot for Median Income

Question: How does median income affect house prices?

plt.figure(figsize=(14, 10))
shap.partial_dependence_plot("MedInc", model.predict, X_background,
                             ice=False, model_expected_value=True,
                             feature_expected_value=True, show=False)
plt.title("Partial Dependence: Impact of Median Income", fontsize=14)
plt.xlabel("Median Income (MedInc)", fontsize=12)
plt.ylabel("Effect on Prediction", fontsize=12)
plt.tight_layout()
plt.show()

What does this plot tell us?
The blue line shows how house prices change as median income increases. It’s almost perfectly straight, meaning the relationship is linear—when median income goes up, house prices go up proportionally.

The gray band around the line is the 95% confidence interval, showing the range of uncertainty in the predictions.

Feature Importance Bar Plot

Question: Which features are the most important overall?

plt.figure(figsize=(14, 10))
shap.summary_plot(shap_values, X_display, plot_type="bar", show=False)
plt.title("Feature Importance: Average Impact on Predictions", fontsize=14)
plt.xlabel("Mean |SHAP Value|", fontsize=12)
plt.tight_layout()
plt.show()

What’s happening here?
This plot ranks features by their average impact on predictions. Longer bars mean more important features. For example, Latitude and Longitude are the most important features in our model.

Force Plot for an Individual Prediction

Question: Why did the model make this specific prediction?

plt.figure(figsize=(16, 6))
shap.force_plot(explainer.expected_value, shap_values[0], X_display.iloc[0],
                matplotlib=True, show=False, text_rotation=0)
plt.title("Force Plot: Detailed Breakdown of an Individual Prediction", fontsize=14)
plt.tight_layout()
plt.show()

What’s going on here?
This plot breaks down the prediction into its individual components. Think of it like a tug-of-war:

• Red bars: Features that push the prediction higher.

• Blue bars: Features that pull the prediction lower.

The base value (the model’s average prediction) is the starting point, and the final prediction is the sum of all the feature contributions.

Wrapping Up

In this post, we:

1. Trained a simple linear regression model.

2. Used SHAP to explain the model’s predictions.

3. Created four visualizations to understand feature importance and individual predictions.

SHAP helps you build trust in your models by making them explainable. It’s like having a flashlight in a dark room—you can finally see what’s going on.

What’s next? Try using SHAP with a more complex model, like a random forest or neural network. You’ll be amazed at the insights you can uncover!

‍🎓Further Learning*

Are you ready to go from zero to building real-world machine learning projects?

Join the AI Learning Hub, a program that will take you through every step of AI mastery—from Python basics to deploying and scaling advanced AI systems.

Why Join?
✔ 10+ hours of content, from fundamentals to cutting-edge AI.
✔ Real-world projects to build your portfolio.
✔ Lifetime access to all current and future materials.
✔ A private community of learners and professionals.
✔ Direct feedback and mentorship.

What You’ll Learn:

• Python, Pandas, and Data Visualization

• Machine Learning & Deep Learning fundamentals

• Model deployment with MLOps tools like Docker, Kubernetes, and MLflow

• End-to-end projects to solve real-world problems

🔗 Join the AI Learning Hub (Lifetime Learning Access)

🔗 Join the AI Learning Hub (Monthly Membership)

Take the leap into AI with the roadmap designed for continuous growth, hands-on learning, and a vibrant support system.

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

⚡Power-Up Corner

You now have a solid understanding of SHAP and how to interpret its outputs—but what if we go a step further? Let’s explore advanced SHAP techniques and some real-world pitfalls to watch out for when applying it in production.

Global vs. Local Interpretability: When SHAP Can Mislead You

SHAP values provide both global and local interpretability, but there’s a catch:

Read more

In this DIY issue, we’ll roll up our sleeves and craft a simple yet effective text sentiment analysis project using the Naive Bayes algorithm. Our mission? Classifying SMS messages as either 'spam' or 'ham' (not spam). This will cover data loading, preprocessing, building the model, and evaluating its performance.

Before starting you can review the theory in this previous issue:

Ready? This is what we will be covering:

• Loading and Exploring the Dataset

• Data Preprocessing

• Splitting Data for Training and Testing

• Text Vectorization with TF-IDF

• Training the Naive Bayes Model

• Evaluating Model Performance

Let's dive in!

1. Loading and Exploring the Dataset

First, we need data! For this project, we’ll be using a dataset containing labeled SMS messages as either spam or ham. Here’s how we get started:

import pandas as pd 

# Load the dataset directly from a URL 
df = pd.read_csv(url, encoding='latin-1') 

# Select relevant columns and rename them for clarity 

df = df[['v1', 'v2']] df.columns = ['label', 'message']

What just happened?

• We loaded data using pandas straight from a URL.

• Kept two important columns: 'label' and 'message'.

• Renamed them for better readability. Quick, simple, and clean!

2. Data Preprocessing

Now, let's prep our data for modeling:

• Convert labels to numerical values: 0 for ham, 1 for spam. This helps our model understand better.

• Check and remove duplicates, ensuring cleaner data for accurate training.

# Transform labels: 0 for 'ham', 1 for 'spam'
df['label'] = df['label'].map({'ham': 0, 'spam': 1})

# Remove duplicates and check for missing values
df = df.drop_duplicates()
print(df.isnull().sum())  # Expect all zeros (no missing values)

How balanced is our data? Quick peek:

print(df['label'].value_counts())

This gives us an idea of how evenly distributed our spam and ham messages are.

3. Splitting Data for Training and Testing

To evaluate our model fairly, we split our data into training and testing sets.

from sklearn.model_selection import train_test_split

# Features and target labels
X = df['message']
y = df['label']

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

What just happened?

• We defined our X and y (features and labels/targets)

• Then we split them into 75% training and 25% testing

4. Text Vectorization with TF-IDF

Text data needs numerical representation for modeling. Enter TF-IDF (Term Frequency-Inverse Document Frequency), which assigns importance to words based on their frequency across messages.

from sklearn.feature_extraction.text import TfidfVectorizer

# Convert text messages to numerical features using TF-IDF
vectorizer = TfidfVectorizer(stop_words='english')
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

Explanation:

• TF-IDF transforms text into a matrix of numerical values, emphasizing unique and relevant words.

• Common stop words (like 'the', 'and') are removed for better focus.

5. Training the Naive Bayes Model

Now for the magic! We’ll use the Multinomial Naive Bayes algorithm, perfect for text classification.

from sklearn.naive_bayes import MultinomialNB

# Initialize and train the model
nb_model = MultinomialNB()
nb_model.fit(X_train_tfidf, y_train)

# Make predictions on the test set
y_pred = nb_model.predict(X_test_tfidf)

Why Multinomial Naive Bayes (MultinomialNB)?
MultinomialNB is tailored for discrete features like word counts or term frequencies, making it ideal for text classification tasks such as spam detection and sentiment analysis. It works by applying Bayes' Theorem under the assumption that features (words) are conditionally independent given the class label.

Breaking it Down Simply:

• P(Cᵢ): The prior probability of class Cᵢ​, indicating how common a class is (e.g., how likely it is that a message is "spam" versus "not spam").

• P(wⱼ|Cᵢ): The probability that word wⱼ​ appears in class Cᵢ​. For instance, if "win" frequently appears in spam messages, P("win"|spam) would be high.

• f(wⱼ, d): The frequency of word wⱼ​ in document d.

What Does the Formula Do?

• It multiplies the prior probability of the class P(Cᵢ) by the probabilities of each word in the document appearing in that class, raised to the power of how often the word appears.

• The higher this product, the more likely the document belongs to that class.

In simple terms, Multinomial Naive Bayes predicts the class of a document by considering how common each word is in that class, adjusted for how often the class itself appears overall. It’s like scoring how "spammy" or "hammy" the words in a message are!

Key Advantages:

• Count-Based Data: Handles features represented by counts or frequencies efficiently (e.g., word occurrence in text).

• High-Dimensional Data: Performs well with large vocabularies and sparse matrices, which are common in text data.

Comparison to GaussianNB:

• Discrete vs. Continuous Data: MultinomialNB works with discrete data, like word counts. In contrast, GaussianNB assumes continuous features that follow a normal distribution, making it more suitable for numerical data.

• Memory Efficiency: MultinomialNB leverages word counts directly, avoiding complex calculations of distribution parameters, which makes it faster for sparse data.

Other Naive Bayes Variants:

• BernoulliNB: Suitable for binary/boolean features (presence/absence of words). Works well when the focus is on word presence rather than frequency.

• GaussianNB: Assumes continuous data with a Gaussian distribution (e.g., features like height, weight).

• ComplementNB: Designed to handle imbalanced classes better, making it effective for datasets where one class is significantly smaller than the other.

6. Evaluating Model Performance

Did it work well? Let’s find out!

from sklearn.metrics import accuracy_score, classification_report

# Calculate and print the accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f'Model Accuracy: {accuracy:.2f}')

# Display a detailed classification report
print(classification_report(y_test, y_pred))

The accuracy score tells us the overall success rate of our model, while the classification report provides deeper insights into precision, recall, and F1-score for spam and ham messages.

🎉 Recap

• Data Loading & Preprocessing: Cleaned and prepared data.

• Text Vectorization: Transformed text to numerical features using TF-IDF.

• Model Training & Evaluation: Trained a Naive Bayes classifier and checked its performance.

Quick, effective, and surprisingly powerful for such a lightweight approach! Stay tuned for more optimizations and advanced techniques in future issues!

If you want the full notebook you can find it at the end of the issue!

It has an additional 3 sections:

1. Data Loading and Exploration

2. Data Preprocessing

3. Data Splitting

4. Text Vectorization

5. Model Training

6. Model Evaluation

7. Hyperparameter Tuning 🆕

8. Model and Vectorizer Saving 🆕

9. Custom Predictions 🆕

⚡Power-Up Corner

To make your text classification model more robust, here are some advanced tips and common pitfalls to watch for:

1. Feature Engineering Beyond TF-IDF:

Read more

In the rapidly evolving world of artificial intelligence, the need for accurate, contextually relevant information has become essential. Traditional language models, while powerful, sometimes generate responses that lack factual accuracy or up-to-date information. This is where Retrieval-Augmented Generation (RAG) comes into play.

This approach combines the best of both worlds: the creativity and fluency of generative models with the factual accuracy of retrieval-based systems.

In this issue of MLPills, we will explore how to build a RAG system using LangChain, a versatile framework that simplifies the integration of language models with external data sources. We'll cover everything from the basics of RAG and LangChain to slightly more advanced optimization techniques and real-world applications.

👨‍💻 We include the notebook and data used at the end of the issue!

What is LangChain?

LangChain is an open-source framework designed to assist developers in creating applications that leverage the capabilities of large language models (LLMs). It provides a set of tools, components, and abstractions that make it easier to integrate language models with various data sources and services, whether for retrieval, processing, or generation tasks.

Key features of LangChain include:

1. Chainability: Allows the creation of complex workflows by chaining together different components.

2. Prompts Management: Offers tools for creating, managing, and optimizing prompts for language models.

3. Memory Integration: Provides mechanisms to give language models short-term and long-term memory capabilities.

4. Agent Framework: Enables the creation of AI agents that can make decisions and take actions.

5. Data Connection: Facilitates easy integration with various data sources and vector stores.

LangChain is particularly useful for building complex AI applications, such as RAG systems, where multiple components need to work together seamlessly. It supports various language models, including those from OpenAI, Hugging Face, and others, making it a flexible choice for developers.

Why Use RAG?

Before proceeding maybe you want to revise what RAG is:

Retrieval-Augmented Generation (RAG) offers several key advantages over traditional language generation techniques:

1. Improved Accuracy: By grounding generative outputs in real-world data, RAG systems produce responses that are more accurate and less prone to hallucinations. This is crucial for applications where factual correctness is paramount.

2. Real-Time Information Retrieval: RAG systems can retrieve up-to-date information from their knowledge base, making them ideal for scenarios where current data is crucial. This is particularly useful in fields like news reporting, market analysis, or providing the latest scientific information.

3. Context-Aware Responses: By using retrieved documents as a basis for generation, RAG systems can provide more contextually relevant answers. This is particularly useful in specialized domains like legal advice, medical information, or technical support, where domain-specific knowledge is critical.

4. Transparency and Explainability: RAG systems can provide the sources of their information, allowing users to verify the generated content. This transparency builds trust and allows for fact-checking.

5. Customizability: The knowledge base used by RAG systems can be easily updated or customized for specific use cases, allowing for greater flexibility compared to traditional language models with fixed training data.

6. Reduced Training Costs: Instead of fine-tuning large language models on domain-specific data (which can be computationally expensive), RAG systems allow for the integration of domain knowledge through the retrieval component.

7. Handling of Long-Context Tasks: RAG systems can effectively handle tasks that require processing or generating long pieces of text by breaking them down into manageable chunks and leveraging relevant information from the knowledge base.

Setting Up Your Environment

Before diving into the code, let's set up the environment necessary for building a RAG system with LangChain.

1. Install Dependencies

To get started, you'll need to install a few Python packages. Open your terminal and run the following command:

pip install langchain openai faiss-cpu python-dotenv langchain-community tiktoken

• LangChain: The core framework we'll be using to build our RAG system.

• OpenAI: To access OpenAI's language models (you can replace this with other supported LLMs if preferred).

• FAISS: A library for efficient similarity search and clustering, which we'll use for document indexing.

• python-dotenv: For managing environment variables, including API keys.

• langchain-community: This library contains a collection of community-contributed utilities, integrations, and modules that extend LangChain's capabilities, enabling easier interaction with a variety of tools and services.

• tiktoken: A library for tokenizing text in a way that's compatible with OpenAI's models, which helps manage token limits and optimize performance when working with large language models.

2. Get API Keys

You'll need API keys for the services you plan to use. For this guide, we'll primarily use OpenAI's API. Follow these steps to obtain and set up your API key:

1. Go to the OpenAI website and sign up for an account if you haven't already.

2. Navigate to the API section and create a new API key.

3. Create a .env file in your project directory and add your API key:

OPENAI_API_KEY=your_api_key_here

4. In your Python script, use the following code to load the environment variables:

from dotenv import load_dotenv 
import os 

load_dotenv() 
openai_api_key = os.getenv("OPENAI_API_KEY")

Step-by-Step Guide to Building a RAG System

Let's walk through the process of building a RAG system using LangChain, step by step.

1. Document Preparation

The first step in creating a RAG system is to prepare the documents that your system will retrieve information from. These documents will serve as the knowledge base for your system.

We will use the text from the following NYTimes article about Climate Change that we manually cleaned and split in subdocuments (TXT files) for the purpose of the explanation.

LangChain supports various document loaders for different file types and sources. Here's an example of how you can load text documents:

from langchain.document_loaders import DirectoryLoader, TextLoader

# Load text files from a directory
loader = DirectoryLoader('./data', glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()

# Process the documents
from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
texts = text_splitter.split_documents(documents)

In this example, we're using the DirectoryLoader to load all .txt files from a data directory. We then use a RecursiveCharacterTextSplitter to split the documents into smaller chunks, which is important for efficient indexing and retrieval.

You can learn more about text chunking in the following issue:

2. Indexing Documents

Once your documents are loaded and processed, the next step is to index them so that they can be efficiently searched when generating responses. We'll use FAISS (Facebook AI Similarity Search) for this purpose.

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS

# Initialize the embeddings
embeddings = OpenAIEmbeddings()

# Create the vector store
vectorstore = FAISS.from_documents(texts, embeddings)

# Save the vector store
vectorstore.save_local("faiss_index")

In this code snippet, we use OpenAI's embeddings to convert the document chunks into vectors, which are then indexed using FAISS. We also save the index locally for future use.

What is a vector store? Here you have more info:

3. Setting Up the Retrieval Mechanism

Now that we have our documents indexed, we can set up the retrieval mechanism. This will allow our system to find and return the most relevant documents based on a user's query.

Read more

Time series data, characterized by observations over sequential time intervals, often exhibits patterns and trends that can be analyzed and forecasted using sophisticated statistical models. One such powerful model is ARIMA (AutoRegressive Integrated Moving Average), which combines autoregressive (AR), differencing (I), and moving average (MA) components to capture the temporal dependencies within the data.

🛠️ DIY

In this article we will show how to train an ARIMA model. We will focus on the results and steps, without showing the code, since it was already introduced in previous issues of MLPills. However, we will share a notebook with all the code you need to train your ARIMA model at the end of the issue!

Step 1: Loading the data and Understanding the process

To begin our analysis, we start by importing essential libraries like numpy and pandas, which facilitate numerical operations and data manipulation, respectively. Loading our time series data using pd.read_csv() allows us to set appropriate timestamps as the index, ensuring accurate time-based analysis.

You can also revise this issue in which we introduced the ARIMA model and its components:

We will follow the so-called Box-Jenkins method:

Step 2: Assessing Stationarity

Before applying ARIMA, it's crucial to ensure that our time series data is stationary. This will be allow us to determine the value of “d”. Stationarity implies that the statistical properties such as mean and variance remain constant over time, which is essential for modeling.

We employ two primary tests for stationarity:

• Augmented Dickey-Fuller (ADF) Test: This test examines whether a unit root is present in the series. A lower p-value (< 0.05) from this test suggests that the series is stationary.

• Kwiatkowski-Phillips-Schmidt-Shin (KPSS) Test: In contrast, this test checks for stationarity around a deterministic trend. A higher p-value (> 0.05) indicates stationarity.

In the example we observe that the initial data is not stationary, we could actually see if just by plotting the graph:

This means that we will need to difference the data, “d” won’t be 0.

Step 3: Preparing the Data - Differencing

The series is not stationary based on the initial tests, so we apply differencing. Differencing involves subtracting the previous observation from the current one, a process denoted by the parameter “d” in ARIMA. This transformation helps stabilize the mean and variance of the series.

This looks way better!

If we perform the stationarity tests we get that it is stationary:

Augmented Dickey-Fuller (ADF) Test:

ADF Statistic: -8.770582534427128
p-value: 2.5274599738010693e-14
Critical Values:
	1%: -3.437
	5%: -2.864
	10%: -2.568
ADF test: Series is stationary

Kwiatkowski-Phillips-Schmidt-Shin (KPSS) Test:

KPSS Statistic: 0.19664101759123429
p-value: 0.1
Lags Used: 11
Critical Values:
	10%: 0.347
	5%: 0.463
	2.5%: 0.574
	1%: 0.739
KPSS test: Series is stationary

Since we had to differenciate once before making our data stationary, we know that the parameter d of the ARIMA model will be equal to 1.

We can proceed with the other two parameters: p and q.

Step 4: Visualizing Correlations - ACF and PACF

To determine the appropriate parameters p (AR) and q (MA) for our ARIMA model, we plot:

• Autocorrelation Function (ACF): This plot shows the correlation between the series and its lagged values, aiding in identifying the MA order (q).

• Partial Autocorrelation Function (PACF): This plot indicates the direct relationship between a lag and the series, helping to determine the AR order (p).

First we can check the ACF plot of the undifferenced data:

You can see that the ACF plot shows a slow decay, which indicates that differencing was indeed necessary.

Now let's plot the PACF and ACF of the differenced data (d=1) to see the difference:

A sharp decline in autocorrelation is observed in both plots, which indicates stationarity, confirming that the selected d value (1) was appropriate.

This will also help us estimate the values for p and q:

• The order of the AR term (p) is typically chosen based on the Partial Autocorrelation Function (PACF) plot, in this case: p = 6

• The order of the MA term (q) is typically chosen based on the Autocorrelation Function (ACF) plot, in our case: q = 3

Step 5: Building and Refining the ARIMA Model

Armed with insights from the ACF and PACF plots, we construct our ARIMA model. Iteratively adjusting parameters based on model diagnostics such as AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion), we aim to find the best-fit model that minimizes these metrics.

We start by checking the model summary:

We observe that the AR terms 4 to 6 are not significant (p-value [P>|z|] < 0.05). So that may indicate that we've overestimated the value of p, let's reduce it to 4 and do the same again:

Now all terms are significant! Great! Let's compare the AIC and BIC to see if they are better (lower) too:

• ARIMA(6,1,3) → AIC = 2805.486, BIC = 2854.553

• ARIMA(4,1,3) → AIC = 2803.404, BIC = 2842.658

It is not a big change but the new model is better! We ideally should repeat this with several combinations of parameters (around our estimation and verifying that the AR and MA terms are significant) until we achieve the optimal AIC and BIC (the lowest).

Here an introduction to these concepts:

Step 6: Evaluating Model Performance

Once the ARIMA model is trained, we proceed to evaluate its performance:

• Residual Analysis: We inspect the residuals to ensure they exhibit no discernible patterns, resembling white noise. Diagnostic checks like Q-Q plots and histogram of residuals verify this assumption.

• Forecasting: Using the fitted model, we generate forecasts for future time points and compare them against actual values. Metrics such as MAE, RMSE, and others quantify the accuracy of our predictions.

In this issue we will focus solely on the first one, residual analysis, leaving the second one for a future issue.

We observe:

• No obvious patterns in the residuals → random noise

• Histogram looks like a normal distribution

• The majority of datapoints lie over the line in the normal Q-Q graph

• All lags (apart from the 0) are within the confidence range (blue-shaded area).

Great news, our model is capturing the data!

In a future issue…

The next steps would be to assess the accuracy of the model:

1. Split your data: Set aside a portion of your data as a test set. Typically, this is done by holding out the last portion of your time series data. For example, you could use the last 20% of your data for testing and the first 80% for training.

2. Refit the model on the training data: Fit your ARIMA model using only the training data. This ensures that your model does not have knowledge of the future values that it is supposed to predict.

3. Forecast: Use your trained model to forecast the values for the period covered by the test set. Compare these forecasted values with the actual values in the test set.

4. Evaluate the forecast accuracy: Calculate accuracy metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE). These metrics will give you an indication of how well your model is performing.

5. Visualize the results: Plot your forecasted values against the actual values to visually inspect how well your model is capturing the trends and patterns in the data.

In conclusion, ARIMA stands as a robust methodology for analyzing and forecasting time series data, offering insights into trends and patterns that are invaluable for decision-making across various domains. By following these systematic steps, analysts and data scientists can leverage ARIMA to derive actionable insights and make informed predictions from time series data.

‍🎓Learn Real-World Machine Learning!*

Do you want to learn Real-World Machine Learning?

Data Science doesn’t finish with the model training… There is much more!

Here you will learn how to deploy and maintain your models, so they can be used in a Real-World environment:

• Elevate your ML skills with "Real-World ML Tutorial & Community"! 🚀

• Business to ML: Turn real business challenges into ML solutions.

• Data Mastery: Craft perfect ML-ready data with Python.

• Train Like a Pro: Boost your models for peak performance.

• Deploy with Confidence: Master MLOps for real-world impact.

🎁 Special Offer: Use "MASSIVE50" for 50% off. Only this time!

Learn Real-World Machine Learning

*Sponsored

🤖 Tech Round-Up

No time to check the news this week?

This week's TechRoundUp comes full of AI news. from Apple's new steps towards AI to Spotify's new in-house Creative Lab.

Let's dive into the latest Tech highlights you probably shouldn’t this week 💥

1️⃣ Apple Unveils New AI Features at WWDC 2024! 🍏🤖

Dive into Apple's latest AI innovations that promise to revolutionize user experience.

From smarter Siri to enhanced privacy controls and real-time language translation, here’s all you need to know.

2️⃣ Job Hunting Made Easy with LinkedIn's New AI Tools 🤖🔍

LinkedIn is leveraging AI to streamline job searches, suggesting roles that fit your skills and experiences perfectly.

Say goodbye to endless scrolling and hello to your dream job!

3️⃣ Meta Pauses AI Training Using European User Data 🌍📊

Amid regulatory pressures, Meta halts plans to train its AI with European data.

This move underscores the growing importance of data privacy and compliance.

4️⃣ Spotify Launches Creative Labs 🎨🎧

Spotify’s new in-house ad agency, Creative Labs, aims to revolutionize audio advertising.

Get ready for more personalized and engaging ads during your music sessions.

5️⃣ Why Apple's AI Approach is Genius in its Simplicity 📱✨

Apple focuses on practical, user-centric AI features rather than flashy gimmicks.

This approach ensures seamless integration into daily use, making tech genuinely useful.

Follow Josep here

📝 Time to practise!

Here you have the notebook with all the code omited in this article:

Read more

Feature importance is the quantification of the predictive power of individual input variables or features in a model. It provides insights into which features contribute most significantly to the model's performance in making accurate predictions or classifications. Understanding feature importance aids feature selection, model interpretation, and identifying critical factors driving the outcomes.

We introduced this concept a month ago:

But today we are going to put it into practice!

The data

We will analyse the most important factors (diagnostic measurements) that allow for an accurate prediction of whether a patient has diabetes. We will use a dataset from the National Institute of Diabetes and Digestive and Kidney
Diseases. We can find several variables, the independent variables
(several medical predictor variables) and the target or dependent variable (whether the person has diabetes or not).

Importance

But before starting, why is Feature Importance important? It is crucial for several reasons:

1. Model Interpretability: Understanding which features are most important in making predictions helps to interpret the model's behavior.

2. Feature Selection: Feature importance helps in identifying the most relevant features for making predictions. In many cases, datasets contain a large number of features, some of which may be irrelevant or redundant.

3. Dimensionality Reduction: By identifying and focusing on the most important features, feature importance analysis aids in reducing the dimensionality of the dataset. This simplifies the model, making it more computationally efficient and less prone to overfitting.

4. Insights for Feature Engineering: Feature importance analysis provides insights into which features contribute most significantly to the target variable. This information can guide feature engineering efforts by highlighting areas where additional domain knowledge or feature transformations might be beneficial.

5. Model Debugging and Diagnosis: Understanding feature importance can help diagnose issues with the model. For instance, if a highly important feature is not available or has incorrect values, it could signal data quality issues or feature engineering mistakes.

6. Communication: Feature importance analysis facilitates communication between data scientists and domain experts. It provides a common ground for discussing which features are driving the predictions and helps in explaining model behavior to stakeholders who may not have a deep understanding of machine learning algorithms.

Random Forest model

We will start by training a Random Forest classifier model. If you remember, decision-tree-based models allow for the estimation of the feature importance based on the assessment of the contribution of each feature to reducing impurity or error during the decision-making process.

This first approach yielded the following results:

It seems that glucose is the main factor, followed by BMI, age and diabetes pedigree function.

Getting this straight from the model is not always possible. That is why there exist techniques that address this issue.

Correlation matrix

One basic approach would be to compute the correlation matrix and see how each feature correlates with the outcome feature:

Let’s simplify this information in a bar chart:

Again we see that glucose is the main predictor, whereas in this case it’s followed by the BMI, age and pregnancies.

The correlation matrix approach is not the best for assessing feature importance because it assumes linear relationships, ignores non-linear connections, struggles with multicollinearity, and cannot capture complex interactions or the combined effects of multiple features on the target variable. Instead, other more advanced techniques are preferred for their ability to handle these complexities and provide more accurate assessments of feature importance.

Permutation importance

Permutation importance is another model-agnostic method that computes feature importance for any model by shuffling the values of each feature one at a time and measuring the resulting change in model performance. The process works by breaking the relationship between each feature and the target variable, then observing the impact on the model’s performance. A higher decrease in performance upon shuffling indicates greater feature importance.

It matches the previous results, but it seems that glucose is way more important than the other predictors.

This technique is better because it captures non-linear relationships and interactions, handles multicollinearity, reflects true predictive power, can be applied to complex models, and is resilient to outliers.

However, it lacks the individualized interpretations, capturing of interactions, and directionality of feature effects that SHAP provides.

SHAP

SHAP (SHapley Additive exPlanations) values offer a unified measure of feature importance that allocates the contribution of each feature to the prediction for every possible combination of features. This method ensures a fair distribution of contributions, as it respects both efficiency and symmetry among features.

Again, glucose is the clear winner, followed by BMI, age and diabetes pedigree function.

However, this method can be computationally expensive and other simpler methods can be just fine depending on your problem.

Recursive Feature Elimination

This is not a feature importance technique, but I think it is interesting for you to also know it. It is an iterative method used for feature selection. The goal of feature selection is to identify and remove unnecessary features from the data that do not contribute, or may even decrease, the predictive performance of the model. RFE achieves this by recursively fitting the model, ranking the features based on their impact on model performance, and removing the least important feature at each step. This process continues until all features have been evaluated and ranked.

However, it’s worth noting that it can also be computationally expensive for models with a large number of features, as it involves repeatedly fitting the model and evaluating its performance. Also, it doesn’t calculate importance, it only ranks features by importance order.

  1: Glucose
  2: BMI
  3: Age
  4: DiabetesPedigreeFunction
  5: BloodPressure
  6: Insulin
  7: Pregnancies
  8: SkinThickness

Again, glucose is the clear winner, followed by BMI and age.

The code

You can find a notebook with all the code at the end of this newsletter issue!

‍🎓Learn Advanced Machine Learning Concepts!*

Do you want to learn more about Model Interpretability and Feature Selection? Ready for a deeper dive?

• Explore feature engineering and feature selection methods

• Explain interpretable and black box models with LIME, Shap, partial dependency plots and more.

Enroll today and take the next step in mastering the world of data science!

Unlock Knowledge!

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

🤖 Tech Round-Up

This week's TechRoundUp comes full of AI news. From Google’s brand new model Gemma to AI giants pact, the future is zooming towards us! 🚀

Let's dive into the latest Tech highlights you probably shouldn’t this week 💥

1️⃣ Google launches Gemma.

It is the next-gen open AI models for responsible development.

Offering tools & models for safer AI applications, it's set to revolutionize AI use responsibly.

2️⃣ Samsung expands Galaxy AI

Samsung is planning to expand AI features to more devices, enhancing user experiences with One UL 6.1 update.

Get ready for smarter, more intuitive device interactions.

3️⃣ Google DeepMind commits to AI safety

It is founding a new organization, focusing on ethical AI development.

A bold step towards responsible AI innovation.

4️⃣ Amazon unveils its largest text-to-speech model

Pushing the boundaries of natural sounding digital voices.

A leap forward in making technology more accessible.

5️⃣ The EU tightens controls

With AI scrutiny, aiming to maintain fair competition and innovation in the AI landscape.

A critical move for the future of AI regulation.

Follow Josep on 𝕏

🛠️ Do It Yourself!

You can practise all these concepts thanks to this notebook. You will need to do some research but don’t worry, next week I’ll unveil the answers!

Practise here:

Read more

Feature engineering is crucial for making time series models better at predicting outcomes. It is all about extracting important information from time-based data. Time series data naturally have patterns and connections, so creating effective features is important for accurate predictions. There are multiple ways of generating these features and each of them is used to capture specific aspects of data. Today we will share the most common ones.

Are you already familiar with this topic and you just want to practice? Check the notebook at the end of this newsletter 👇👇

1. Date/Time-Related Features: These features are derived from the date-time value of each observation. They can include the year, month, day, hour, minute, second, day of the week, and whether the day is a weekend or a holiday.

   1. These features can be useful when there is a trend or seasonality in your data that corresponds to these periods.

   2. For example, retail sales might increase during weekends or holidays or website traffic might peak during certain hours.

2. Lag Features: Lag features are values at prior time steps. They can help capture the temporal dependencies in the data.

   1. These features can be useful when past values of a series are useful for predicting its future values.

   2. For example, in stock price prediction, the price of a stock in the past few days can be useful for predicting its price tomorrow.

3. Rolling Window Features: These features are statistical measures like mean, median, standard deviation, etc., over a sliding or rolling window of time periods.

   1. These features can be useful when you want to capture local trends and patterns in your data.

   2. For example, a rolling mean of temperature readings can smooth out daily fluctuations and highlight longer-term trends. Also, it can show the sales inertia of the period of interest.

4. Expanding Window Features: These are similar to rolling window features but the window size increases with time.

   1. These features can be useful when you want to capture all the past information up to the current point.

   2. For example, the expanding mean of a stock’s price can give you its average price since the beginning of the time series, i.e. the historic price average.

5. Domain-Specific Features: These are features that are specific to the problem at hand.

   1. These features can be useful when you have domain knowledge that can help you create informative features.

   2. For example, in a stock price prediction problem, features like the company’s earnings, the sector’s performance, etc., can be used.

6. Time Since an Event: This feature measures the time that has passed since a particular event occurred.

   1. This can be useful in scenarios where the occurrence of an event significantly impacts the time series data.

   2. For example, in predicting website traffic, an event could be a marketing campaign, and the feature could be the time since the campaign started.

7. Autoregressive Features: These are based on the idea that past values have an influence on current values. An autoregressive feature of order p would use the last p values. This is similar to lag features but instead of using the raw values, we use the values predicted by an autoregressive model.

   1. These features can be useful when modeling time series data with dependencies on its own past, such as predicting stock prices where historical trends impact future values.

   2. For example, when predicting stock prices, an autoregressive feature of order 3 would consider the last three days' predicted values to capture short-term trends.

8. Difference Features: These features represent the difference between consecutive values in the time series. Differences can highlight trends or abrupt changes in the data.

   1. These features can be useful when identifying and analyzing patterns that emerge as changes or trends between consecutive observations.

   2. For example, in weather forecasting, difference features might highlight sudden temperature changes, aiding in the prediction of weather patterns.

9. Exponential Moving Averages: Similar to moving averages, exponential moving averages give more weight to recent observations.

   1. These features can be useful when modelling data with evolving patterns or trends that change more rapidly over time.

   2. For example, in predicting user engagement on a website, exponential moving averages can give more weight to recent user activity, providing insights into current trends.

10. Seasonal Features: Seasonal features capture recurring patterns in the data related to specific seasons or periods. You can create binary features indicating whether the observation falls within a particular season or period.

    1. These features can be useful when predicting phenomena influenced by recurring cycles, like sales patterns associated with holidays or special events.

    2. For example, in retail sales forecasting, seasonal features could help account for increased sales during holiday seasons, improving the accuracy of sales predictions.

11. Cyclical Features: Some time series data exhibits cyclical patterns that may not align with standard date-related features. Creating cyclical features can help capture such patterns.

    1. These features can be useful when modelling time series data with recurring but non-linear patterns, enabling the model to better capture and understand cyclic variations.

    2. For example, in predicting electricity consumption throughout the day, cyclical features involving sine and cosine transformations of the hour could account for the daily cycle, helping the model adapt to the fluctuating demand patterns.

Do you want to put all this into practice? Check the notebook at the end of the newsletter 👇👇

🤖 Tech Round-Up

No time to check the news this week?

This week's TechRoundUp comes full of AI news. From AI in literature to robotics in manufacturing, the future is zooming towards us! 🚀

Let's dive into the latest Tech highlights you shouldn’t miss this week 💥

1️⃣ 𝗖𝗵𝗮𝘁𝗚𝗣𝗧'𝘀 𝗟𝗶𝘁𝗲𝗿𝗮𝗿𝘆 𝗟𝗲𝗮𝗽 📚

Japan's literary world embraces AI!

A renowned author openly uses ChatGPT for writing assistance.

She says it's not just a tool; it's a creative partner! 🤖✍️

2️⃣ 𝗦𝗮𝗺𝘀𝘂𝗻𝗴 𝗦24: 𝗔𝗜-𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗱 𝗘𝗹𝗲𝗴𝗮𝗻𝗰𝗲 📱

The upcoming Galaxy S24 is all set to dazzle with Google Gemini's AI power.

Expect smarter features and seamless experiences!

Will it be the first phone to have native LLM? 🤔

3️⃣ 𝗣𝗶𝗻𝗲𝗰𝗼𝗻𝗲'𝘀 𝗣𝗶𝗼𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗣𝗮𝘁𝗵 🌲

Pinecone revamps its vector database with a serverless architecture, promising more efficient data management and scalability.

A game-changer in data processing!

4️⃣ 𝗕𝗠𝗪'𝘀 𝗥𝗼𝗯𝗼𝘁𝗶𝗰 𝗥𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻 🤖🚗

BMW's South Carolina plant gears up for a futuristic makeover with Figure's humanoid robots.

Say hello to a new era of automated efficiency!

5️⃣ 𝗗𝗲𝗲𝗽𝗠𝗶𝗻𝗱'𝘀 𝗚𝗲𝗼𝗺𝗲𝘁𝗿𝗶𝗰 𝗚𝗲𝗻𝗶𝘂𝘀 🔢

DeepMind's AlphaGeometry AI cracks complex geometry problems, rivaling Olympiad champs.

A breakthrough in AI problem-solving and mathematical reasoning!

Follow Josep on 𝕏

Learn Advanced Machine Learning Concepts!*

Have you outgrown introductory courses? Ready for a deeper dive?

• Explore feature engineering and feature selection methods

• Discover tactics for optimizing hyperparameters and addressing imbalanced data

• Master fundamental machine learning methods and their Python application

Enroll today and take the next step in mastering the world of data science!

Unlock Knowledge!

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

🛠️ Do It Yourself!

Time for you to play with the code!

I share with you the notebook with almost everything you need. Your task is to make it work and get the results I shared in this newsletter. I provide you with some hints. The best way of learning is by checking the documentation.

Contact me at david@mlpills.dev if you need any help!

Read more

In the last DIY, we introduced the concept of Exploratory Data Analysis (EDA). This is a fundamental step in Data Science. One of the key components of the EDA is finding out about the data distribution.

Here is a reminder of what EDA is:

Let’s inspect the data distribution in more depth!

What is data distribution and why do we need to be aware of it?

It refers to how data points are spread across various values, showing the frequency of each value. Being aware of data distributions is crucial because it informs the choice of statistical methods and algorithms for analysis, ensuring accurate and meaningful insights. Different distributions, like normal, skewed, or uniform, can significantly impact data interpretation and the effectiveness of predictive models.

You can check here some of the most common distributions in data science.

Check the distribution of your data

Here you’ll find out how you can investigate how data in each of your features are distributed.

Let’s start by importing the essential libraries for data analysis. These are:

• Pandas is a powerful data manipulation library that provides data structures for efficient data handling.

• Matplotlib and Seaborn are visualization libraries used for creating various types of plots.

• SciPy is employed for its statistical functions, including normality tests.

Histograms and Density Plots

Histograms provide a visual representation of the distribution of each variable. By specifying the number of bins, we can control the granularity of the representation. Density Plots, overlaid on histograms, provide a smoothed representation of the data distribution.

Box Plots

Box Plots visually summarize the distribution of each variable by displaying the median, quartiles, and potential outliers. They are handy for identifying the spread of data and understanding central tendencies.

Summary Statistics

Summary Statistics provide a snapshot of key metrics such as mean, standard deviation, and quartiles. This step helps gain a quick understanding of the central tendencies and variability within the dataset.

Skewness and Kurtosis

Skewness measures the asymmetry of the distribution, indicating whether it is skewed to the left or right. Kurtosis provides insights into the tails of the distribution, helping to understand the peakedness.

Normality Tests with Q-Q Plots

Normality Tests, such as the Shapiro-Wilk test, assess whether a variable follows a normal distribution. Q-Q Plots visually inspect the degree of agreement between the observed data and the theoretical quantiles of a normal distribution.

Scatter Plots

Scatter Plots provide a visual representation of relationships between pairs of variables. The pairplot from Seaborn generates scatter plots for all combinations of variables, offering insights into potential correlations and patterns.

These technical details provide a deeper understanding of each step in the Exploratory Data Analysis process, elucidating the specific functions and methodologies used for data exploration and visualization.

You can check how to build each of them in the 🪐Jupyter notebook you’ll find at the end of the newsletter!

🤖 Tech Round-Up

This second week's TechRoundUp comes full of AI news.

Let's discover the top AI & tech highlights of the week!

From Amazon to Google search, the future of tech is unfolding before our eyes.

1️⃣ 𝗚𝗼𝗼𝗴𝗹𝗲'𝘀 𝗚𝗲𝗺𝗶𝗻𝗶 𝗔𝗜: 𝗔 𝗚𝗮𝗺𝗲 𝗖𝗵𝗮𝗻𝗴𝗲𝗿 𝗼𝗿 𝗝𝘂𝘀𝘁 𝗛𝘆𝗽𝗲?

Google unveils Gemini, their latest LLM 🆕

With advanced capabilities, Gemini is set to beat GPT-4 and revolutionize the way we interact with AI.

2️⃣ 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗕𝗶𝗻𝗴'𝘀 𝗚𝗣𝗧-4 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻

Microsoft Bing amps up search with GPT-4 🤖

Their new 'Deep Search' digs deeper, delivering comprehensive answers to complex queries.

3️⃣ 𝗠𝗲𝘁𝗮 𝗮𝗻𝗱 𝗜𝗕𝗠'𝘀 𝗔𝗜 𝗔𝗹𝗹𝗶𝗮𝗻𝗰𝗲

Meta and IBM's AI Alliance is here! 🤝

This collaboration fosters open innovation in AI, uniting a wide range of sectors for responsible, inclusive AI advancement

4️⃣ 𝗔𝗪𝗦 𝗡𝗲𝗽𝘁𝘂𝗻𝗲 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀

A new era for data! 💥

It combines vector search and graph data, offering unique insights and tackling high-dimensional data and relationships with ease

5️⃣ 𝗩𝗥-𝗕𝗮𝘀𝗲𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗶𝗻 𝗞𝗼𝗿𝗲𝗮

VR meets construction safety training in Korea! 🚧

Researchers propose a machine learning model using VR and biometrics for personalized training.

Follow Josep on 𝕏

Learn Advanced Machine Learning Concepts!*

Have you outgrown introductory courses? Ready for a deeper dive?

• Explore feature engineering and feature selection methods

• Discover tactics for optimizing hyperparameters and addressing imbalanced data

• Master fundamental machine learning methods and their Python application

Enroll today and take the next step in mastering the world of data science!

Unlock Knowledge!

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

🛠️ Do It Yourself!

Time for you to play with the code!

I will share with you the notebook with almost everything you need. Your task is to make it work and get the results I shared in this newsletter. I provide you with some hints. The best way of learning is by checking the documentation.

Enjoy and good luck!

Read more

💊 Pill of the week: Build a regression ML model

We are going to build a very simple model that will allow us to predict the salary of a person based on their years of experience and age.

You can get the notebook at the end of the issue 👇

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

Read dataset. You can download it here. It is a very simple dataset with three columns:

• Years of experience

• Age

• Salary: this is the target, the values we want to predict. During the training, this will work as the labels, to teach the model how to predict it based on the other features.

We can read this dataset (CSV file) easily using pandas read_csv.

df = pd.read_csv('/kaggle/input/salary-data-with-age-and-experience/Salary_Data.csv')

It is always good to have a look at your data to verify that it was correctly imported. You can check the top 5 or 10 rows (df.head(5)), the bottom 5 or 10 rows (df.tail(5)) or simply a random sample (df.sample(5)). Here we checked 5 random rows of the dataset:

df.sample(5)

Exploratory Data Analysis

Exploratory Data Analysis (EDA) is a process used for investigating your data to discover patterns, anomalies, relationships, or trends using statistical summaries and visual methods. This approach is essential for understanding the data's underlying structure and characteristics before applying more formal statistical or Machine Learning methods. Some key points that are normally checked are:

1. Distribution of Data: Assessing the distribution of data (e.g., normal, skewed) using histograms, box plots, and summary statistics helps understand the central tendency and variability.

2. Missing Values: Identifying and addressing missing data is crucial, as it can significantly affect analyses. Techniques include imputation, deletion, or understanding the reasons for missingness.

3. Outliers: Detecting and examining outliers to understand their impact on the dataset and deciding how to handle them (e.g., removal, transformation).

4. Correlations: Analyzing correlations between variables using correlation coefficients and scatter plots to identify relationships and potential dependencies.

5. Patterns and Trends: Looking for patterns, trends, or anomalies in the data, which can be visualized using line graphs, bar charts, or time-series analysis.

6. Group Comparisons: Comparing metrics across different groups (e.g., categories, time periods) to identify significant differences or similarities.

7. Data Type Assessment: Understanding the types of data (numerical, categorical, ordinal) and their appropriate treatment in analysis.

8. Data Quality Assessment: Evaluating data quality to identify errors or inconsistencies that may need correction.

9. Visual Exploration: Employing various visualization tools (like heatmaps, pair plots) to intuitively understand complex relationships in the data.

The purpose of this issue is to introduce the general Data Science process, so we will not deep dive into the EDA. However, we can do a very basic one. For example, we can check the datatypes, missing values and number of records using the "info" method:

df.info()

We can see that there are no null or missing values, all columns contain numbers (floats or integers) and there is a total of 30 rows. I know, this is very simple and not a real-world example! Normally this won’t be that easy…

Before continuing… why are missing values problematic? Check this if you want to find out more:

We can also use the method "describe" to check some descriptive statistics. For example, the average value, the maximum and minimum, etc.

df.describe()

This is a good way of quickly checking the data distribution, but it would be good to also do some plots to understand it better. But it is fine for today… We will do this in future issues.

Learn Advanced Machine Learning Concepts!*

Have you outgrown introductory courses? Ready for a deeper dive?

• Explore feature engineering and feature selection methods

• Discover tactics for optimizing hyperparameters and addressing imbalanced data

• Master fundamental machine learning methods and their Python application

Enroll today and take the next step in mastering the world of data science!

Unlock Knowledge!

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

Split data in train and test

You need to train your model using only some data (training set), and then check if it behaves properly with the other part that you left aside (testing set). This is to check whether your model will behave correctly in the real world, with data that hasn't been seen before.

First, we divide the dataset into the label y (salary) and features X (years of experience and age):

X = df.iloc[:,:-1]
y = df.iloc[:,-1]

Now we need to get the training set and the testing set. We need the majority of the data in the training set because it is the one the model will use to learn. For example, we can use 80% of the data for training and 20% for testing. This needs to be adapted depending on the data. If you have a lot of data you could reduce the testing set because even a small percentage will include many samples.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=1)

Train the model

We will use Linear Regression, which is the simplest model. I introduced it here:

You can find more details here:

First, let's instantiate (load) the model. We could also specify some parameters if we wished. You can get more details here.

lr = LinearRegression()

Fit the training data. This is the model training, which is the process of teaching the machine learning algorithm to make predictions or decisions by learning from a dataset. This involves using a dataset to tune the parameters of the model so that it can accurately generalize from the training data to new, unseen data.

lr = lr.fit(X_train, y_train)

After fitting the training data, we should assess the performance of the model on the testing set. This is to make sure that it can accurately generalise to unseen data (testing set).

lr.score(X_test, y_test)

For our dataset, it achieved an R-squared of 0.77. R-squared or coefficient of determination of the prediction is a statistical measure that is commonly used to assess the performance of regression models.

Read more

In this issue, we will talk about base models for Time Series forecasting.

What are base models?

In Time Series Analysis and Forecasting, a base model is often a simple model used as a benchmark to compare the performance of more complex models. Here are several basic or naive methods that are commonly used as base models in time series forecasting:

1. Naive Forecast (NF)

• This method simply uses the last observed value as the forecast for all future time points. It is naive because it does not consider any other information from the past.

• Formula:

  

2. Simple Average (SA)

• This method calculates the average of all past observations and uses this value as the forecast. It assumes that future values will revolve around the average of past values.

• Formula:

  

  where 𝑇 is the number of observations.
  

3. Moving Average (MA)

• This method averages the last n observations to forecast the next value. It smoothens short-term fluctuations and highlights longer-term trends or cycles.

• Formula:

  

4. Exponential Smoothing (ES)

• This method gives more weight to the most recent observations and less weight to the older ones, with weights decreasing exponentially.

• Formula:

  

  where α is the smoothing parameter.

Do you want to take your ML skills to the next level? Discover Train In Data:

Self-passed online learning courses, featuring in-depth modules on feature engineering and selection, working with imbalanced data, optimizing hyperparameters, time series forecasting and more. A great complement to MLPills to take your skills to the next level!

Check Train In Data's courses

*Sponsored: by purchasing any of their courses you would also be supporting MLPills.

5. Seasonal Naive Forecast (SNF)

• This method assumes that the future value will be equal to the last observed value from the same season.

• Formula:

  

  where s is the seasonal period and m is the forecast horizon.
  

6. Drift Method (DM)

• This method extrapolates the line connecting the first and the last observation to forecast future values.

• Formula:

  

  where h is the forecast horizon and T is the number of observations.
  

7. Random Walk (RW)

• This method assumes that changes in the time series are random, and future values are unpredictable, being equal to the last observed value plus a random error.

• Formula:

  

  where ϵₜ​ is a white noise error term.

8. Mean Reversion (MR)

• This method assumes that the series will revert to its mean over time, forecasting future values based on the mean and the last observed value.

• Formula:

  

  where μ is the mean and β is the reversion coefficient.

How to Choose a Base Model?

• Simplicity: Base models are typically simple and easy to understand.

• Data Characteristics: Consider the characteristics of the data, such as seasonality and trend, when choosing a base model.

• Performance Metric: Evaluate the base model using appropriate performance metrics like MAE, RMSE, or MAPE to set a benchmark for more sophisticated models.

Here is a simple diagram to help you choose the best benchmark model. You could also select multiple ones and keep the one that shows a better performance.

Conclusion

These naive methods serve as a starting point and help in establishing a baseline performance. Any sophisticated model should ideally perform significantly better than these naive models to be considered useful.

👉 Check the next section to put what you've learned into practice!

This could perfectly be a Data Science interview question. You can check additional questions on the website!

If you like it, subscribe for free to support us:

Subscribe now

🛠️ Do It Yourself!

Now that you know the theory about base models and when you should use each of them, it’s time to apply it!

How does it work?

• 📜I will share a notebook with some guided initial steps.

• 📌I will ask you some tasks that you should complete.

• 🎯I will share the outcome so you can check if you did well or not!

Now it’s your turn. Let’s play!

I want you to apply each of the five techniques I previously introduced:

1. 🖨Naive Forecast - Difficulty ⭐

2. ⚖️Simple Average - Difficulty ⭐

3. 🪟Moving Average - Difficulty ⭐

4. 🥑Exponential Smoothing - Difficulty ⭐⭐

5. 🏖Seasonal Naive Forecast - Difficulty ⭐⭐

6. 🛝Drift Method - Difficulty ⭐⭐

7. 🎲Random Walk - Difficulty ⭐

8. 🎢Mean Reversion - Difficulty ⭐

I provide you with the average monthly temperature in London for the last 40 years (1982 - 2022). Create eight base models to see which one behaves better when forecasting the last 2 years. You will then be able to use this model as the benchmark when you train a more complex model like SARIMA, Holt-Winters, LSTM…

You should get something like this:

Here you can find a Kaggle notebook with everything you need below:

Read more

💊 Pill of the week

In this issue, we will deal with encoding.

What are categorical variables?

Categorical variables are those that take on a limited, fixed number of distinct categories or classes. They are often human-understandable labels such as "red," "blue," "apple," "banana," etc., and don't have a natural numerical representation. To convert these variables into a format that can be provided to machine learning algorithms, we use encoding.

What is encoding?

Encoding is the process of converting data from one form to another. In the context of machine learning and data science, encoding often refers to the transformation of categorical variables into a numerical format that can be easily used by algorithms. Most machine learning algorithms require numerical input and output variables, hence the need for encoding.

The main goal of encoding in machine learning is to translate the data into a form that is useful and understandable by the algorithm, without distorting the characteristic properties and relationships in the data.

It's worth noting that improper encoding can introduce biases or inaccuracies in the model, so choosing the right encoding method is crucial.

Main techniques

1. Label Encoding

In this technique, each unique category is mapped to an integer starting from 0. It does not assume any relationship of order or magnitude between the categories. Categories are numbered arbitrarily.

When to Use: Best suited for ordinal data where the order matters but can be used for nominal data when the algorithm can handle it correctly (e.g., decision trees).

Pros:

• Simple to implement.

• Does not increase dimensionality.

Cons:

• Because it introduces ordinality, it can lead to a misleading sense of magnitude or ordinal relationship.

2. One-Hot Encoding

It converts each unique category into a new binary column of 1 or 0.

When to Use: For nominal categories where no ordinal relationship exists.

Pros:

• Easy to use and interpret.

• No ordinal relationships are introduced.

Cons:

• Dimensionality: Introduces as many new columns as there are unique values in the original column, which can explode dimensionality.

• Multicollinearity: The encoding can introduce multicollinearity which can be problematic for certain algorithms (like linear regression).

3. Ordinal Encoding

It maps each unique category to an integer based on the inherent ordinal nature of the category.

When to Use: For ordinal data where the order of categories is important.

Pros:

• Keeps the ordinal relationship.

• Does not increase dimensionality.

Cons:

• Difficult to set up correctly.

• Subjectivity: Determining the correct order of categories can be subjective and/or data-dependent.

• Misinterpretation: Incorrect ordering may lead the model to learn inaccurate patterns from the data.

4. Frequency or Count Encoding

Categories are replaced with their frequency or count in the data set.

When to Use: When there are too many categories and one-hot encoding increases dimensionality too much.

Pros:

• Effective for high cardinality features.

• Does not increase dimensionality.

Cons:

• Loses information about the categories.

• Different categories might end up having the same frequency, causing a collision.

5. Target Encoding

Categories are replaced with the mean of the target variable for that category.

When to Use: When the category has some correlation with the target variable. Be cautious of data leakage.

Pros:

• Can capture information within the category that can aid in prediction.

• Useful for high cardinality features.

Cons:

• Prone to data leakage: If not done correctly, target encoding can result in data leakage that can inflate the performance metrics.

• Overfitting: This encoding is sensitive to outliers and can result in overfitting if the number of categories is small.

⚠️ Each encoding technique has its own advantages and disadvantages, and the choice of which to use often depends on the specific problem and the type of data you're working with. Always remember to test out different approaches and validate their effectiveness using cross-validation or a separate validation set.

👉 Check the next section to put what you've learned into practice!

This could perfectly be a Data Science interview question. You can check additional questions on the website!

If you like it, subscribe for free to support us:

Subscribe now

🛠️ Do It Yourself!

Now that you know the theory about feature scaling and when you should use each of the techniques, it’s time to apply it!

How does it work?

• 📜I will share a notebook with some guided initial steps.

• 📌I will ask you some tasks that you should complete.

• 🎯I will share the outcome so you can check if you did well or not!

Now it’s your turn. Let’s play!

I want you to apply each of the five techniques I previously introduced:

1. 🎲Label Encoding - Difficulty ⭐

2. 🧮One-Hot Encoding - Difficulty ⭐⭐

3. 🔢Ordinal Encoding - Difficulty ⭐

4. ⏱Frequency Encoding - Difficulty ⭐⭐

5. ⚖️Target Encoding - Difficulty ⭐⭐

Here you can find a Kaggle notebook with everything you need:

Read more

💊 Pill of the week

In this issue, we will deal with feature scaling. Are you an advanced user and you don’t care about the theory? Go straight to the Kaggle notebook that I have prepared:

Go to the Kaggle notebook!

Otherwise, stay with me, I will introduce you to the concept.

What is data or feature scaling?

Data preprocessing is a crucial step in the machine learning pipeline, ensuring that the dataset is ready for training. One essential aspect of data preprocessing is feature scaling, which involves adjusting the range and distribution of the data. Feature scaling techniques, such as normalization and standardization, ensure that each attribute contributes equally to the algorithm's performance, thereby enhancing the predictive model's accuracy and efficiency. Both normalization and standardization aim to re-scale features, but they do so in different ways and are suitable for different types of data and machine learning algorithms.

Normalization

Normalization is a technique used to scale the features (or variables) of a dataset to a similar range. The goal is to transform each feature to lie in a certain interval, usually [0,1], making it easier for algorithms to interpret these features.

Normalization ensures that each feature contributes equally to the computation of distances or gradients. This is why it is especially important for distance-based algorithms like k-nearest neighbours (KNN), or when your features have different units or vastly different scales.

Standardization

Standardization, unlike normalization, transforms the features in a way that the resulting distribution has a mean of 0 and a standard deviation of 1. It doesn't bound values to a specific range.

This is especially useful for algorithms that assume the input variables to have a Gaussian distribution.

Introduction to the Different Techniques

• Min-Max Scaling:

  • This is the simplest form of normalization. Each feature is scaled linearly between the minimum and maximum value to a range [0,1].

  • Use it when the distribution of the feature is not Gaussian and you need values in a bounded interval. However, this method is sensitive to outliers.

  • Mathematically, the simplest form of normalization for a feature x is calculated as:

    

• Z-score Normalization (Standardization):

  • Here, features are scaled so that they have the properties of a standard normal distribution with μ=0 and σ=1.

  • Use it when the algorithm assumes that the distribution of your features is Gaussian. This method is also useful as a general technique when you don't know the distribution of your feature and you're not particularly concerned about robustness to outliers.

  • Mathematically, standardization for a feature x is calculated as follows. Where μ is the mean and σ is the standard deviation of the feature.

    

• Decimal Scaling:

  • In this method, the data is scaled by moving the decimal point of values of each feature.

  • Useful when the range of the dataset is unknown or could vary widely, but this method is rarely used in practice.

  • To scale the feature x, you find d, which is the smallest integer such that max(|x|) / 10ᵈ < 1, and then scale x by 10⁻ᵈ.

    

• Robust Scaling:

  • In this method, the median and the interquartile range are used for scaling, making it robust to outliers.

  • Use it when the data contains many outliers and you want to be robust against them.

  • It is calculated by subtracting the median and dividing by the interquartile range of the feature (IQR):

    

• Log Transformation:

  • In this method, the logarithm is applied to each of the values of the feature.

  • Use it when the feature is highly skewed and transforming it could make it easier to model.

  • The feature x is transformed using the natural logarithm:

    

👉 Check the next section to put what you've learned into practice!

This could perfectly be a Data Science interview question. You can check additional questions on the website!

If you like it, subscribe for free to support us:

Subscribe now

🛠️ Do It Yourself!

Now that you know the theory about feature scaling and when you should use each of the techniques, it’s time to apply it!

How does it work?

• 📜I will share a notebook with some guided initial steps.

• 📌I will ask you some tasks that you should complete.

• 🎯I will share the outcome so you can check if you did well or not!

Now it’s your turn. Let’s play!

I want you to apply each of the five techniques I previously introduced:

1. 🌡Min-Max Scaling - Difficulty ⭐

2. 🔔Z-score Normalization (Standardization) - Difficulty ⭐

3. ⛳️Decimal Scaling - Difficulty ⭐

4. ⛰️Robust Scaling - Difficulty ⭐

5. 🎢Log Transformation - Difficulty ⭐⭐

Here you can find a Kaggle notebook with everything you need!

Go to the Kaggle notebook!

Starting from this data:

You should be able to get the following outcomes:

• 🌡Min-Max Scaling

• 🔔Z-score Normalization (Standardization)

• ⛳️Decimal Scaling

• ⛰️Robust Scaling

And starting from this other data source:

You should be able to get the following outcome:

• 🎢Log Transformation

That would mean that your data has been successfully scaled!

You will find the answer next week!