This is the notes that I took when walking through this lecture.


I did two important modifications,

  1. I used a local LLM served by Ollama instead of chatGPT in OpenAI.
  2. I used a new version (0.2.7) and fixed a lot compatible issues as the original course was for (0.1.x).

Models, Prompts, and Parsers

Using an LLM via Ollama

from langchain_community.llms import Ollama

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
llm.invoke("hello, what is your name?")

# output
'Hello! I\'m LLaMA, a large language model ...'

Chatting with a template

from langchain.prompts import ChatPromptTemplate, PromptTemplate
from langchain.output_parsers import ResponseSchema
from langchain.output_parsers import StructuredOutputParser

Here is an example where we use LLM to Process customer emails.

template_string = """Translate the text \
that is delimited by triple brackticks \
into a style that is {style}. \
text: ```{text}```
"""
prompt_template = ChatPromptTemplate.from_template(template_string)

customer_email = """\
Arrr, I be fuming that me blender lid \
flew off and splattered me kitchen walls \
with smoothie! And to make matters worse, \
the warranty don't cover the cost of \
cleaning up me kitchen. I need yer help \
right now, matey!
"""
style = "American English in a calm and respectful tone"

customer_messages = prompt_template.format_messages(
    style=style,
    text=customer_email
)
print(llm.invoke(customer_messages))

Generated output:

Here's the translation:

"I'm really upset about ..."

Use LLM to get JSON output

We want to extract result in the format of Python dictionary, which corresponds to a JSON dataset.

{
  "gift": False,
  "delivery_days": 5,
  "price_value": "pretty affordable!"
}

We can use LangChain to achieve this

customer_review = """\
This leaf blower is pretty amazing.  It has four settings:\
candle blower, gentle breeze, windy city, and tornado. \
It arrived in two days, just in time for my wife's \
anniversary present. \
I think my wife liked it so much she was speechless. \
So far I've been the only one using it, and I've been \
using it every other morning to clear the leaves on our lawn. \
It's slightly more expensive than the other leaf blowers \
out there, but I think it's worth it for the extra features.
"""

review_template = """\
For the following text, extract the following information:

gift: Was the item purchased as a gift for someone else? \
Answer True if yes, False if not or unknown.

delivery_days: How many days did it take for the product \
to arrive? If this information is not found, output -1.

price_value: Extract any sentences about the value or price,\
and output them as a comma separated Python list.

Format the output as JSON with the following keys:
gift
delivery_days
price_value

text: {text}
"""

prompt_template = ChatPromptTemplate.from_template(review_template)

Let’s use the local LLM to get the response.

messages = prompt_template.format_messages(text=customer_review)
response = llm.invoke(messages)
print(response)

The generated result:

"""
Here is the extracted information in JSON format:

```json
{
    "gift": True,
    "delivery_days": 2,
    "price_value": ["It's slightly more expensive ..."]
}
```

Let me explain my reasoning:
...
"""

We need to use output_parsers to extract dict objects from the output string.

gift_schema = ResponseSchema(
    name="gift",
    description="Was the item purchased as a gift for someone else?\
        Answer `true` if yes, `false` if not or unknown."
)
delivery_days_schema = ResponseSchema(
    name="delivery_days",
    description="How many days did it take for the product to arrive?\
        If this information is not found, output `-1`."
)
price_value_schema = ResponseSchema(
    name="price_value", 
    description="Extract any sentences about the value or \
        price, and output them as a comma separated Python list."
)
response_schemas = [
    gift_schema, delivery_days_schema, price_value_schema
]

output_parser = StructuredOutputParser.from_response_schemas(
    response_schemas
)
format_instruction = output_parser.get_format_instructions(False)
print(format_instruction)

The output should be a markdown code snippet formatted in the following schema, including the leading and trailing “```json” and “```”:

{
    "gift": string  // Was the item purchased...
    "delivery_days": string  // How many days did ...
    "price_value": string  // Extract any sentences ...
}

Now we include the format_instruction to the prompt template

review_template_json = """\
For the following text, extract the following information:

gift: Was the item purchased as a gift for someone else? \
Answer True if yes, False if not or unknown.

delivery_days: How many days did it take for the product\
to arrive? If this information is not found, output -1.

price_value: Extract any sentences about the value or price,\
and output them as a comma separated Python list.

text: {text}

{format_instructions}

No not respond anything other than a valid json snippet!
"""

prompt_template = ChatPromptTemplate.from_template(
    review_template_json
)

messages = prompt_template.format_messages(
    text=customer_review,
    format_instructions=format_instruction  #  new format
)

response = llm.invoke(messages)
print(response)

Here is the output response.

{
    "gift": true,
    "delivery_days": 2,
    "price_value": [
        "slightly more expensive ..."
    ]
}

We can use the output_parser to extract the Python dictionary object from the generated output:

output_parser.parse(response)

here is the output results.

{
    'gift': True,
    'delivery_days': 2,
    'price_value': [
        "slightly more expensive ...."
    ]
}

Memory

There are several types of memory in LangChain

We can manipulate the memroy via the folowing methods.

from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from langchain.memory import ConversationBufferWindowMemory
from langchain.memory import ConversationTokenBufferMemory
from langchain.memory import ConversationSummaryBufferMemory

from langchain_core.runnables.history import RunnableWithMessageHistory

ConversationBufferMemory

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationBufferMemory()
conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True,
)

conversation.predict(input="Hi, my name is Andrew")

Here is the first response

"Nice to meet you too, Andrew! My name is Ada,..."

We can ask another question

conversation.predict(input="What is 1+1?")

Here is the second response.

"A classic question! The answer to 1+1 is indeed... 2! ..."

Finally we ask our name again.

conversation.predict(input="What is my name?")

The chatbot can answer correctly because of the memory.

"Easy one! Your name is Andrew, correct? ..."

Let’s check the memory buffer – it will be the overall conversation history.

print(memory.buffer)

"""
Human: Hi, my name is Andrew
AI: Nice to meet you too, Andrew! ...
Human: What is 1+1?
AI: A classic question! ...
Human: What is my name?
AI: Easy one! Your name is Andrew, correct? ...
"""

clear memory

We can use the following command to clear the history in the memory.

memory.load_memory_variables({})

add extra memory

we can use save_context to add content maually to the memory (prompt history)

memory.save_context(
    {"input": "Hi"}, 
    {"output": "What's up"}
)
                    
conversation.predict(input="What's up with you?")

here is the response

"Ha ha, nice one, Andrew! ..."

Window Memory

We can also use a “windowed” memory where only the latest/newest conversations are remembered.

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationBufferWindowMemory(k=1)
conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True,
)

Here is our first conversation. I ask the LLM to respond me in Chinese and to be as concise as possible.

conversation.predict(
	input="你好,你会说中文吗?请用中文回答,并且尽量简洁。"
)

'哈喽!我可以说中文 ...'  # LLM response

Here is our second conversation.

conversation.predict(input="我的名字叫 XXX,你呢")

'哈喽 XXX!我的名称是 LLaMA ...'  # LLM response

Since we set the memory window k = 1, the next conversation will not include the initial chat. The chat will not be in Chinese anymore and it won’t be concise.

conversation.predict(input="你可以教我西班牙语吗?")

"¡Hola! Of course, I'd be happy to help ..."  # LLM response

Memory with a Fixed Token size

We need tiktoken and transformers to use this memory buffer. Run

pip install tiktoken transformers
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)

memory = ConversationTokenBufferMemory(
    llm=llm, max_token_limit=500
)

conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True,
)

conversation.predict(
    input="你好,你会说中文吗?请用中文回答,并且尽量简洁。",
)

'你好!我可以说中文 ...'  # LLM response

conversation.predict(
    input="你可以用中文教我西班牙语吗?"
)

'I would be happy to help you ...'  # LLM response

Summary in the Memory

We can compress the memory:

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)

memory = ConversationSummaryBufferMemory(
    llm=llm, max_token_limit=100
)

conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True,
)

Here are the conversations.

conversation.predict(
    input="你好,你会说中文吗?请用中文回答,并且尽量简洁。"
)

'你好!我可以说中文 ...'  # LLM response

conversation.predict(
    input="哇塞你真厉害!"
)

'哈哈,谢谢你的夸奖! ... '  # LLM response

conversation.predict(
    input="你可以用中文教我西班牙语的基本语法吗?"
)

# LLM response
"""
'System: Current summary:\n\nThe human greets the AI and asks if it can speak Chinese. The AI responds by saying hello, confirming its ability to communicate in Chinese, and noting that it has learned a large amount of Chinese data, allowing for an exchange in Chinese. The human then introduces themselves as XXX and asks about the significance of their name. The AI analyzes the name\'s components, including "Yang" as an old Chinese surname and "Yu Shi" as a traditional form of Chinese poetry, suggesting that the name may express appreciation for nature or artistic pursuit.\n\nNew lines of conversation:\n\nHuman: 你可以用中文教我西班牙语的基本语法吗?\nAI:\n\n\nNew summary:\n\nThe human greets the AI and asks if it can speak Chinese. The AI responds by saying hello, confirming its ability to communicate in Chinese, and noting that it has learned a large amount of Chinese data, allowing for an exchange in Chinese. The human then introduces themselves as Yang Yu Po and asks about the significance of their name. The AI analyzes the name\'s components, including "Yang" as an old Chinese surname and "Yu Shi" as a traditional form of Chinese poetry, suggesting that the name may express appreciation for nature or artistic pursuit. The human then asks if the AI can teach them basic Spanish grammar in Chinese, and the AI is happy to help.\n\nPlease go ahead and continue the conversation! 😊'
"""

Chains in LangChain

Here we explain the concenpt of chains from langchain.

In this section, we also read a data in the file Data.csv.

import pandas as pd
from langchain.chains import (
    LLMChain, SimpleSequentialChain, SequentialChain
)

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
df = pd.read_csv('Data.csv')

LLMChain

The most simple chain. The API changed in 2024. for old LangChain Versions <= 0.1, using the following format

chain = LLMChain(llm=llm, prompt=prompt)
print(chain.run(product))

The following code exhibits the newer version (0.2+).

prompt = ChatPromptTemplate.from_template(
    "What is the best name to describe "
    "a company that makes {product}?"
    "Give me a concise and terse answer please."
)

chain = prompt | llm

product = "Queen Size Sheet Set"

print(chain.invoke(product))

"Royal Slumber Co."  # LLM Response

Simple Sequential Chain

It works for the case where each individual chains have a single input and a single output.

llm = Ollama(model=LLM_MODEL)

prompt_name = ChatPromptTemplate.from_template(
    "What is the best name to describe \
    a company that makes {product}?"
    "Give me a concise and terse answer please."

)
prompt_desc = ChatPromptTemplate.from_template(
    "Write a 20 words description for the following"
    "company:{company_name}"
    "Give me a concise and terse answer please."
)

chain_name = LLMChain(llm=llm, prompt=prompt_name)
chain_desc = LLMChain(llm=llm, prompt=prompt_desc)

chain = SimpleSequentialChain(
    chains=[chain_name, chain_desc], verbose=True
)

chain.invoke(product)['output']

# Output
"RoyalRests Inc. provides luxurious services ..." #LLM Response

Sequential Chain

Sequential Chain allow use to construct complicated workflows

flowchart LR
    A(French Review) --> B[English Review]
    B --> S[English Summary]
    A --> L[Language]
    L --> R(French Response)
    S --> R
prompt_tran = ChatPromptTemplate.from_template(
    "Translate the following review to english:"
    "\n\n{review}"
    "\nbe concise and do not add anything other than the review text."
)
prompt_summ = ChatPromptTemplate.from_template(
    "Can you summarize the following review in 1 sentence:"
    "\n\n{review_english}"
    "\nbe concise and do not add anything other than the review text."
)
prompt_lang = ChatPromptTemplate.from_template(
    "What language is the following review:\n\n{review}"
    "\nbe concise and do not add anything other than the language name."
)
prompt_resp = ChatPromptTemplate.from_template(
    "Write a follow up response to the following "
    "summary in the specified language:"
    "\n\nSummary: {summary}\n\nLanguage: {language}"
    "\nbe concise and do not add anything other than the response."
)

chain_tran = LLMChain(
    llm=llm, prompt=prompt_tran, output_key="review_english"
)
chain_summ = LLMChain(
    llm=llm, prompt=prompt_summ, output_key="summary"
)
chain_lang = LLMChain(
    llm=llm, prompt=prompt_lang, output_key="language"
)
chain_resp = LLMChain(
    llm=llm, prompt=prompt_resp, output_key="response"
)

chain = SequentialChain(
    chains=[chain_tran, chain_summ, chain_lang, chain_resp],
    input_variables=["review"],
    output_variables=["review_english", "summary", "response"],
    verbose=True
)

review = df.Review[4]

outputs = chain.invoke(review)
for term, content in outputs.items():
    print(f"## {term}")
    print(content + "\n")

The output is too long so we omit it.

Router Chain

flowchart LR

I(Input) --> R(Router)
R --> P(Physics)
R --> C(Computer Science)
R --> D(default)
from langchain.chains.router import MultiPromptChain
from langchain.chains.router.llm_router import LLMRouterChain,RouterOutputParser

Here we create the chains for the router to choose from.

physics_template = """You are a very smart physics professor. \
You are great at answering questions about physics in a concise\
and easy to understand manner. \
When you don't know the answer to a question you admit\
that you don't know.
Try to be as concise as possible.

Here is a question:
{input}"""

computerscience_template = """ You are a successful computer scientist.\
You have a passion for creativity, collaboration,\
forward-thinking, confidence, strong problem-solving capabilities,\
understanding of theories and algorithms, and excellent communication \
skills. You are great at answering coding questions. \
You are so good because you know how to solve a problem by \
describing the solution in imperative steps \
that a machine can easily interpret and you know how to \
choose a solution that has a good balance between \
time complexity and space complexity. 
Try to be as concise as possible.

Here is a question:
{input}"""

prompts = [
    {
        "name": "physics", 
        "description": "Good for answering questions about physics", 
        "prompt_template": physics_template
    },
    {
        "name": "computer science", 
        "description": "Good for answering computer science questions", 
        "prompt_template": computerscience_template
    }
]

destination_chains = {}
for p_info in prompts:
    name = p_info["name"]
    prompt_template = p_info["prompt_template"]
    prompt = ChatPromptTemplate.from_template(template=prompt_template)
    chain = LLMChain(llm=llm, prompt=prompt)
    destination_chains[name] = chain


default_prompt = ChatPromptTemplate.from_template("{input}")
default_chain = LLMChain(llm=llm, prompt=default_prompt)

Here we prepare for the router chain

destinations = [f"{p['name']}: {p['description']}" for p in prompts]
destinations_str = "\n".join(destinations)
print(destinations_str)

# Output
"""
physics: Good for answering questions about physics
computer science: Good for answering computer science questions
"""

Now we use this very long prompt to guild the LLM.

MULTI_PROMPT_ROUTER_TEMPLATE = r"""Given a raw text input to a \
language model select the model prompt best suited for the input. \
You will be given the names of the available prompts and a \
description of what the prompt is best suited for. \
You may also revise the original input if you think that revising\
it will ultimately lead to a better response from the language model.

<< FORMATTING >>
Return a markdown code snippet with a JSON object formatted to look like:
\`\`\`json
{
    "destination": string \ name of the prompt to use or "DEFAULT"
    "next_inputs": string \ a potentially modified version of the original input
}
\`\`\`

REMEMBER: "destination" MUST be one of the candidate prompt \
names specified below OR it can be "DEFAULT" if the input is not\
well suited for any of the candidate prompts.
REMEMBER: "next_inputs" can just be the original input \
if you don't think any modifications are needed.

<< CANDIDATE PROMPTS >>
{destinations}

<< INPUT >>


<< OUTPUT (remember to include the ```json)>>
"""

router_template = MULTI_PROMPT_ROUTER_TEMPLATE.format(
    destinations=destinations_str
)

router_prompt = PromptTemplate(
    template=router_template,
    input_variables=["input"],
    output_parser=RouterOutputParser(),
)

Now we create the chain with the router.

router_chain = LLMRouterChain.from_llm(llm, router_prompt)

chain = MultiPromptChain(
    router_chain=router_chain, 
    destination_chains=destination_chains, 
    default_chain=default_chain, verbose=True
)

# LLM will choose Physics
chain.invoke("What is Percus Yevick equation?")

# LLM will use Computer Science
chain.invoke("what is the travelling salesman problem")

# Console Output
"""
→ Entering new MultiPromptChain chain...
    ✦ computer science: {
        'input': "Is there a specific solution ...?"
    }
→ Finished chain.
"""

# LLM response
"""
A classic problem!

For TSP, I'd recommend exploring dynamic programming
approaches like Christofides Algorithm or 2-Opt
Heuristics. These methods balance time and space
complexity well.
"""

# LLM will use Computer Science
chain.invoke("what is NGS?")

# Console Output
"""
→ Entering new MultiPromptChain chain ...
    computer science: {
        'input': 'What is Next Generation Sequencing (NGS)?'
    }
→ Finished chain.
"""

# LLM Response
"""
Next-Generation Sequencing (NGS) refers to the technology
used in DNA sequencing, which allows for rapid and
cost-effective analysis of an individual's entire genome
or specific regions of interest. NGS enables the simultaneous
analysis of millions of DNA fragments, resulting in faster
turnaround times and higher resolution than traditional
Sanger sequencing methods.
"""

QA for Documents

from langchain.document_loaders import CSVLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.vectorstores import DocArrayInMemorySearch
from langchain_community.embeddings import OllamaEmbeddings
from langchain.chains import RetrievalQA
from IPython.display import display, Markdown

Overview

To do a RAG we need

Creating the Vector DB

EMBEDDING_MODEL = "all-minilm"
file = 'OutdoorClothingCatalog_1000.csv'
loader = CSVLoader(file_path=file)
index = VectorstoreIndexCreator(
    embedding=OllamaEmbeddings(model=EMBEDDING_MODEL),
    vectorstore_cls=DocArrayInMemorySearch
).from_loaders([loader])

Generate response given a query

LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)

query ="""\
Please list all your shirts with sun protection
in a table in markdown and summarize each one.
"""

response = index.query(query, llm=llm)

display(Markdown(response))

Here are the shirts with sun protection listed in a table:

Shirts Description Sun Protection
Sun Shield Shirt High-performance sun shirt for UV ray protection SPF 50+ (blocks 98% of sun’s harmful rays)
Tropical Breeze Shirt Lightweight, breathable long-sleeve UPF shirt for superior sun protection SPF 50+ (blocks 98% of sun’s harmful rays)
Women’s Tropical Tee, Sleeveless Five-star sleeveless button-up shirt with SunSmart UPF 50+ rating SPF 50+ (blocks 98% of sun’s harmful rays)
Girls’ Beachside Breeze Shirt, Half-Sleeve Rash guard-style swim shirt with built-in UPF 50+ protection SPF 50+ (blocks 98% of sun’s harmful rays)

Summary: All the shirts listed have high-performance fabric with SPF 50+ sun protection, blocking 98% of the sun’s harmful UV rays. They are suitable for outdoor activities such as swimming, fishing, and travel to protect skin from damage.

RAG Step-by-Step

Load the Documents

loader = CSVLoader(file_path=file)
docs = loader.load()
# docs[0] print
"""
Document(
  metadata={
    'source': 'OutdoorClothingCatalog_1000.csv', 'row': 0
  },
  page_content="..."
)
"""

Create the Vector Database

embeddings = OllamaEmbeddings(model=EMBEDDING_MODEL)
embed = embeddings.embed_query("Hi my name is Yushi")

len(embed) # -> 384
embed[:5]  # [-0.4055, 0.2655, 0.1268, 0.4260, -0.5726]

# vector database in memory
db = DocArrayInMemorySearch.from_documents(
    docs, embeddings
)

Generate the Response

llm = Ollama(model=LLM_MODEL, verbose=True)
retriever = db.as_retriever()

query = "Please suggest a shirt with sunblocking"
docs = db.similarity_search(query, k=4)
qdocs = "".join(
    [docs[i].page_content for i in range(len(docs))]
)

response = llm.invoke(
    f"{qdocs} Question: Please list all your \
shirts with sun protection in a table in markdown"
)

display(Markdown(response))

Here is the list of shirts with sun protection in a table format using Markdown:

Shirt Name Description Sun Protection
Sun Shield Shirt Block the sun, not the fun - our high-performance sun shirt is guaranteed to protect from harmful UV rays. UPF 50+ rated, blocks 98% of the sun’s harmful rays
Tropical Breeze Shirt Beat the heat in this lightweight, breathable long-sleeve men’s UPF shirt, offering superior SunSmart protection from the sun’s harmful rays. UPF 50+ rated, blocks 98% of the sun’s harmful rays
Women’s Tropical Tee, Sleeveless Our five-star sleeveless button-up shirt has a fit to flatter and SunSmart protection to block the sun’s harmful UV rays. UPF 50+ rated, blocks 98% of the sun’s harmful rays
Men’s Plaid Tropic Shirt, Short-Sleeve Our Ultracomfortable sun protection is rated to UPF 50+, helping you stay cool and dry. UPF 50+ rated, blocks 98% of the sun’s harmful rays

Note: All shirts have a UPF (Ultraviolet Protection Factor) rating of 50+, which means they block at least 95% of UVB rays and at least 90% of UVA rays.

Retrieval QA Chain

The RetrievalQA chain can achieve identical RAG results.

qa_stuff = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type="stuff",  # "stuffs" all documents into context
    retriever=db.as_retriever(), 
    verbose=True
)

query = "Please list all your shirts with sun protection"
query += " in a table in markdown."
response = qa_stuff.invoke(query)

# console output
"""
→ Entering new RetrievalQA chain
→ Finished chain.
"""

display(Markdown(response['result']))

Here are the shirts with sun protection listed in a table:

Shirt Name Sun Protection
Sun Shield Shirt by [name] UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays)
Women’s Tropical Tee, Sleeveless UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays)
Tropical Breeze Shirt UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays)
Girls’ Beachside Breeze Shirt, Half-Sleeve UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays)

Let me know if you need anything else!

Retrievel Methods

Stuff

put all retrieved chunks into the context. Then call LLM once. The most common method.

flowchart LR
A[Documents] --> B(chunk)
A --> C(chunk)
A --> D(chunk)
B --> L[LLM]
C --> L
D --> L  --> R[Response]

Map Reduce

pass each chunk and query to LLM, then use LLM to process the responses. The second most common method.

flowchart LR
A[Documents] --> B(chunk) --> LB[LLM]
A --> C(chunk) --> LC[LLM]
A --> D(chunk) --> LD[LLM]
LB --> L[LLM]
LC --> L
LD --> L  --> R[Response]

Refine

Builds the answer iteratively.

flowchart LR
A[Documents] --> B(chunk) --> LB[LLM]
A --> C(chunk) --> LC[LLM]
A --> D(chunk) --> LD[LLM]
LB --> LC
LC --> LD
LD --> R[Response]

Map Rerank

flowchart LR
A[Documents] --> B(chunk) --> LB[LLM] --> SB(Score 40)
A --> C(chunk) --> LC[LLM] --> SC(Score 60) --> R[Response]
A --> D(chunk) --> LD[LLM] --> SD(Score 32)

LLM Evaluation

General Instruction

  1. Visualisation: to understand the input/output data.
  2. Create examples for evaluation.
  3. Use LLM to evaluate.

Here is a Step by Step guide.

Setting Up

file = 'OutdoorClothingCatalog_1000.csv'
loader = CSVLoader(file_path=file)
data = loader.load()

index = VectorstoreIndexCreator(
    embedding=OllamaEmbeddings(model=EMBEDDING_MODEL),
    vectorstore_cls=DocArrayInMemorySearch
).from_loaders([loader])
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)

qa = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type="stuff", 
    retriever=index.vectorstore.as_retriever(), 
)

To evaluate the result, we should create standard “Question” + “Answer” examples.

Creating Examples Manually

We can generate such examples (Question + Answer) by

  1. Observing the data.
  2. Asking a question ourselves.
  3. Answer the question ourselves.

Here is an example

# data[10].page_content

: 10
name: Cozy Comfort Pullover Set, Stripe
description: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.

Size & Fit
- Pants are Favorite Fit: Sits lower on the waist.
- Relaxed Fit: Our most generous fit sits farthest from the body.

Fabric & Care
- In the softest blend of 63% polyester, 35% rayon and 2% spandex.

Additional Features
- Relaxed fit top with raglan sleeves and rounded hem.
- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.

Imported.

Here is the manually crafted example.

examples = [
    {
        "query": "Do the Cozy Comfort Pullover Set have side pockets?",
        "answer": "Yes"
    },
    {
        "query": "What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?",
        "answer": "The DownTek collection"
    }
]

Generating Examples with LLMs

the following code was copied from

langchain/evaluation/qa/generate_chain.py

So that our code will be compatible with the new interface for langchain 0.2+

from langchain.output_parsers.regex import RegexParser

qa_gen_parser = RegexParser(
    regex=r"QUESTION: (.*?)\n+ANSWER: (.*)", output_keys=["query", "answer"]
)

qa_gen_template = """\
You are a teacher coming up with questions to ask on a quiz.\
Given the following document, please generate a question and\
answer based on that document.

Example Format:
<Begin Document>
...
<End Document>
QUESTION: question here
ANSWER: answer here

These questions should be detailed and be based explicitly on\
information in the document. Begin!

<Begin Document>
{doc}
<End Document>"""

qa_gen_prompt = PromptTemplate(
    input_variables=["doc"],
    template=qa_gen_template,
)

We now build the QAGenerateChain with the runable interface.

For old LangChain version (<= 0.1), we can use the following code

qa_gen = QAGenerateChain.from_llm(llm)

new_examples = qa_gen.apply_and_parse(
   [{"doc": t} for t in data[:5]]
)
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)

qa_gen = qa_gen_prompt | llm | qa_gen_parser

new_examples = qa_gen.batch(
    [{"doc": t} for t in data[:5]]
)

# new examples
[{
    'query': "What type of material is ... ?",
    'answer': 'Soft canvas material.'
    }, {
    'query': 'What percentage of recycled ... ?',
    'answer': '94%'
    }, {
    'query': "What feature of this toddler ... ?",
    'answer': 'The UPF 50+ rated fabric.'
    }, {
    'query': "What percentage of the ... ?",
    'answer': '82%'
    }, {
    'query': 'What technology does EcoFlex ... ?',
    'answer': 'TEK O2 technology.'
}]

Manually Evaluate

examples += new_examples

qa.invoke(examples[0]["query"])

# output
{
  'query': 'Do the Cozy Comfort Pullover Set have side pockets?',
  'result': 'According to ... the answer is:\n\nYes ...'
}

examples[0]["answer"]  # Output: 'Yes'

LLM assisted evaluation

The default prompt can be found in the file

langchain/evaluation/qa/eval_prompt.py

Here we edited the prompt so that the output is good for llama3.

from langchain.evaluation.qa import QAEvalChain

grade_template = """You are a teacher grading a quiz.
You are given a question, the student's answer, and the true answer, 
and are asked to score the student answer as either CORRECT or INCORRECT.

Example Input Format:
-----
QUESTION: question here
STUDENT ANSWER: student's answer here
TRUE ANSWER: true answer here
-----

Example Output:
CORRECT or INCORRECT

Grade the student answers based ONLY on their factual accuracy.
Ignore differences in punctuation and phrasing between the student answer and true answer.
It is OK if the student answer contains more information than the true answer,
as long as it does not contain any conflicting statements.

Return the final grade. The only options are between CORRECT and INCORRECT.

Here are the actual input data:

QUESTION: {query}
STUDENT ANSWER: {result}
TRUE ANSWER: {answer}
"""
grade_prompt = PromptTemplate(
    input_variables=["query", "result", "answer"],
    template=grade_template
)

eval_chain = QAEvalChain.from_llm(llm, prompt=grade_prompt)

predictions = qa.batch(examples)
graded_outputs = eval_chain.evaluate(examples, predictions)

# graded_outputs content
[{'results': 'CORRECT'},
 {'results': 'CORRECT'},
 {'results': 'CORRECT'},
 {'results': 'CORRECT'},
 {'results': 'CORRECT'},
 {'results': 'CORRECT'},
 {'results': 'CORRECT'}]


for i, eg in enumerate(examples):
    print(f"Example {i}:")
    print("Question:         " + predictions[i]['query'])
    print("Real Answer:      " + predictions[i]['answer'])
    print("Predicted Answer: " + predictions[i]['result'])
    print("Predicted Grade:  " + graded_outputs[i]['results'])
    print()

# concole output
"""
Example 0:
Question:         Do the Cozy Comfort Pullover Set have side pockets?
Real Answer:      Yes
Predicted Answer: According to the ...
Predicted Grade:  CORRECT

Example 1:
Question:         What collection is ... ?
Real Answer:      The DownTek collection
Predicted Answer: The Ultra-Loft 850 Stretch ...
Predicted Grade:  CORRECT

Example 2:
Question:         What type of material is ... ?
Real Answer:      Soft canvas material.
Predicted Answer: According to the description ...
Predicted Grade:  CORRECT

Example 3:
Question:         What percentage of recycled ... ?
Real Answer:      94%
Predicted Answer: According to the context ... the answer is 94%.
Predicted Grade:  CORRECT

Example 4:
Question:         What feature of this toddler's swimsuit ... ?
Real Answer:      The UPF 50+ rated fabric.
Predicted Answer: Based on the context provided, the answer ...
Predicted Grade:  CORRECT

Example 5:
Question:         What percentage of the swimtop's body ...?
Real Answer:      82%
Predicted Answer: Based on the context, I found ...
Predicted Grade:  CORRECT

Example 6:
Question:         What technology does EcoFlex 3L ... ?
Real Answer:      TEK O2 technology.
Predicted Answer: According to the context, EcoFlex ...
Predicted Grade:  CORRECT
"""

Debug

We can use the following setup to use the debug mode and see the exact input/ouput for the LLM.

from langchain.globals import set_debug
set_debug(True)

response = qa.invoke(examples[0]["query"])

# Massive amount of information will be printed.

set_debug(False)  # turn off debug mode

Agents

from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain.agents import load_tools, create_react_agent, AgentExecutor
from langchain_core.pydantic_v1 import BaseModel, Field

Creating a tool calling LLM

Detail: Ollama model

reference

We have to install langchain-experimental to use the ollama model.

pip install langchain_experimental

I also changed the default system prompt a bit so that llama3 is more likely to return a json file.

Code

LLM_MODEL = "llama3"
TOOL_PROMPT = """
You have access to the following tools:

{tools}

You must always select one of the above tools and respond with only a 
JSON object matching the following schema:

{
  "tool": <name of the selected tool>,
  "tool_input": <parameters for the selected tool, matching the tool's JSON schema>
}

DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
"""

llm = OllamaFunctions(
    model=LLM_MODEL,
    temperature=0.0,
    verbose=True,
    tool_system_prompt_template=TOOL_PROMPT
)

class GetWeather(BaseModel):
    """Get the current weather in a given location"""
    location: str = Field(
        ..., description="The city and state, e.g. San Francisco, CA"
    )

llm_with_tools = llm.bind_tools([GetWeather])

ai_message = llm_with_tools.invoke(
    "what is the weather like in San Francisco",
)

# ai_message.tool_calls 
[{
  'name': 'GetWeather',
  'args': {'location': 'San Francisco, CA'},
  'id': 'call_4eb652ffba9547b48ab49cc301d50385'
}]
"""

Creating an Agent

For old langchain version, the agent can be created via the following code.


from langchain.agents import initialize_agent, AgentType


tools = load_tools(["llm-math","wikipedia"], llm=llm)
agent= initialize_agent(
    tools, llm, 
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    handle_parsing_errors=True,
    verbose=True
)

Here we use the newer interface for langchain 0.2+.

LLM_MODEL = "llama3"
TOOL_PROMPT = """
You have access to the following tools:

{tools}

You must always select one of the above tools and respond with\
only a JSON object matching the following schema:

{
  "tool": <name of the selected tool>,
  "tool_input": <parameters for the selected tool, matching the tool's JSON schema>
}

DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
"""

llm = OllamaFunctions(
    model=LLM_MODEL,
    temperature=0.0,
    verbose=True,
    tool_system_prompt_template=TOOL_PROMPT
)

react_template = '''
Answer the following questions as best you can.
You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Follow the follwoing rules strictly no matter what

- DO NOT ADD EXTRA DESCRIPTION FOR the action and action input!
- DO NOT REPEAT ANY INFORMATION!
- DO NOT REPEAT THE QUESTION!
- FOLLOW THE FORMAT REQUIREMENT!

Begin!

Question: {input}
Thought:{agent_scratchpad}
'''

react_prompt = PromptTemplate.from_template(react_template)

tools = load_tools(["wikipedia", "wikipedia"], llm=llm)
agent= create_react_agent(llm, tools, prompt=react_prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

We can interact with the agent via the invoke method.

question = "Who is Joe Biden"
result = agent_executor.invoke({"input": question})

# console output
"""
⇨ Entering new AgentExecutor chain...
  ⇨ Here's my response:
      Thought: I need to find information about Joe Biden.
      Action: wikipedia
      Action Input: Joe Biden
        ▶ Page: Joe Biden
        Summary: Joseph Robinette Biden ...
        ▶ Page: Family of Joe Biden
        Summary: Joe Biden, the 46th and ...
        ▶ Page: Presidency of Joe Biden
        summary: Joe Biden's tenure as the 46th president ...
      Final Answer: Joe Biden is an American politician ...
⇨ Finished chain.
"""

Custom Tool

from langchain.agents import tool
from datetime import date

we create a new tool called timer.

@tool
def timer(text: str) -> str:
    """Returns todays date, use this for any \
    questions related to knowing todays date. \
    The input should always be an empty string, \
    and this function will always return todays \
    date - any date mathmatics should occur \
    outside this function."""
    return str(date.today())

We use the tool like this

tools = load_tools(["wikipedia"], llm=llm)
tools.append(timer)
agent= create_react_agent(llm, tools, prompt=react_prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

question = "what is the time now?"
result = agent_executor.invoke(
    {"input": question}
)

# console output
"""
⇨ Entering new AgentExecutor chain...
    Here's my attempt at answering your question:
    Thought:
    Action: timer
    Action Input: 2024-07-19 Here's the answer:
    Thought: I now know the final answer
    Final Answer: 2024-07-19
⇨ Finished chain.
"""