- Models, Prompts, and Parsers
- Memory
- Chains in LangChain
- QA for Documents
- Retrievel Methods
- LLM Evaluation
- Agents
This is the notes that I took when walking through this lecture.
I did two important modifications,
- I used a local LLM served by Ollama instead of chatGPT in OpenAI.
- I used a new version (0.2.7) and fixed a lot compatible issues as the original course was for (0.1.x).
Models, Prompts, and Parsers
Using an LLM via Ollama
from langchain_community.llms import Ollama
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
llm.invoke("hello, what is your name?")
# output
'Hello! I\'m LLaMA, a large language model ...'
Chatting with a template
from langchain.prompts import ChatPromptTemplate, PromptTemplate
from langchain.output_parsers import ResponseSchema
from langchain.output_parsers import StructuredOutputParser
Here is an example where we use LLM to Process customer emails.
template_string = """Translate the text \
that is delimited by triple brackticks \
into a style that is {style}. \
text: ```{text}```
"""
prompt_template = ChatPromptTemplate.from_template(template_string)
customer_email = """\
Arrr, I be fuming that me blender lid \
flew off and splattered me kitchen walls \
with smoothie! And to make matters worse, \
the warranty don't cover the cost of \
cleaning up me kitchen. I need yer help \
right now, matey!
"""
style = "American English in a calm and respectful tone"
customer_messages = prompt_template.format_messages(
style=style,
text=customer_email
)
print(llm.invoke(customer_messages))
Generated output:
Here's the translation:
"I'm really upset about ..."
Use LLM to get JSON output
We want to extract result in the format of Python dictionary, which corresponds to a JSON dataset.
{
"gift": False,
"delivery_days": 5,
"price_value": "pretty affordable!"
}
We can use LangChain to achieve this
customer_review = """\
This leaf blower is pretty amazing. It has four settings:\
candle blower, gentle breeze, windy city, and tornado. \
It arrived in two days, just in time for my wife's \
anniversary present. \
I think my wife liked it so much she was speechless. \
So far I've been the only one using it, and I've been \
using it every other morning to clear the leaves on our lawn. \
It's slightly more expensive than the other leaf blowers \
out there, but I think it's worth it for the extra features.
"""
review_template = """\
For the following text, extract the following information:
gift: Was the item purchased as a gift for someone else? \
Answer True if yes, False if not or unknown.
delivery_days: How many days did it take for the product \
to arrive? If this information is not found, output -1.
price_value: Extract any sentences about the value or price,\
and output them as a comma separated Python list.
Format the output as JSON with the following keys:
gift
delivery_days
price_value
text: {text}
"""
prompt_template = ChatPromptTemplate.from_template(review_template)
Let’s use the local LLM to get the response.
messages = prompt_template.format_messages(text=customer_review)
response = llm.invoke(messages)
print(response)
The generated result:
"""
Here is the extracted information in JSON format:
```json
{
"gift": True,
"delivery_days": 2,
"price_value": ["It's slightly more expensive ..."]
}
```
Let me explain my reasoning:
...
"""
We need to use output_parsers to extract dict objects from the output string.
gift_schema = ResponseSchema(
name="gift",
description="Was the item purchased as a gift for someone else?\
Answer `true` if yes, `false` if not or unknown."
)
delivery_days_schema = ResponseSchema(
name="delivery_days",
description="How many days did it take for the product to arrive?\
If this information is not found, output `-1`."
)
price_value_schema = ResponseSchema(
name="price_value",
description="Extract any sentences about the value or \
price, and output them as a comma separated Python list."
)
response_schemas = [
gift_schema, delivery_days_schema, price_value_schema
]
output_parser = StructuredOutputParser.from_response_schemas(
response_schemas
)
format_instruction = output_parser.get_format_instructions(False)
print(format_instruction)
The output should be a markdown code snippet formatted in the following schema, including the leading and trailing “```json” and “```”:
{
"gift": string // Was the item purchased...
"delivery_days": string // How many days did ...
"price_value": string // Extract any sentences ...
}
Now we include the format_instruction to the prompt template
review_template_json = """\
For the following text, extract the following information:
gift: Was the item purchased as a gift for someone else? \
Answer True if yes, False if not or unknown.
delivery_days: How many days did it take for the product\
to arrive? If this information is not found, output -1.
price_value: Extract any sentences about the value or price,\
and output them as a comma separated Python list.
text: {text}
{format_instructions}
No not respond anything other than a valid json snippet!
"""
prompt_template = ChatPromptTemplate.from_template(
review_template_json
)
messages = prompt_template.format_messages(
text=customer_review,
format_instructions=format_instruction # new format
)
response = llm.invoke(messages)
print(response)
Here is the output response.
{
"gift": true,
"delivery_days": 2,
"price_value": [
"slightly more expensive ..."
]
}
We can use the output_parser to extract the Python dictionary object from the generated output:
output_parser.parse(response)
here is the output results.
{
'gift': True,
'delivery_days': 2,
'price_value': [
"slightly more expensive ...."
]
}
Memory
There are several types of memory in LangChain
- Buffer Memory (
ConversationBufferMemory) - Buffer Window Memory (
ConversationBufferWindowMemory) - Token Buffer Memory (
ConversationTokenBufferMemory) - Summary Buffer Memory (
ConversationSummaryBufferMemory) - Vector Data Memory (store result in a vector DB)
- Entity Memory (use LLM to learn details about entities)
We can manipulate the memroy via the folowing methods.
load_memory_variables(): to replace memory with a python dictionary.buffer: to get the content in the memory.
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from langchain.memory import ConversationBufferWindowMemory
from langchain.memory import ConversationTokenBufferMemory
from langchain.memory import ConversationSummaryBufferMemory
from langchain_core.runnables.history import RunnableWithMessageHistory
ConversationBufferMemory
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationBufferMemory()
conversation = ConversationChain(
llm=llm,
memory=memory,
verbose=True,
)
conversation.predict(input="Hi, my name is Andrew")
Here is the first response
"Nice to meet you too, Andrew! My name is Ada,..."
We can ask another question
conversation.predict(input="What is 1+1?")
Here is the second response.
"A classic question! The answer to 1+1 is indeed... 2! ..."
Finally we ask our name again.
conversation.predict(input="What is my name?")
The chatbot can answer correctly because of the memory.
"Easy one! Your name is Andrew, correct? ..."
Let’s check the memory buffer – it will be the overall conversation history.
print(memory.buffer)
"""
Human: Hi, my name is Andrew
AI: Nice to meet you too, Andrew! ...
Human: What is 1+1?
AI: A classic question! ...
Human: What is my name?
AI: Easy one! Your name is Andrew, correct? ...
"""
clear memory
We can use the following command to clear the history in the memory.
memory.load_memory_variables({})
add extra memory
we can use save_context to add content maually to the memory (prompt history)
memory.save_context(
{"input": "Hi"},
{"output": "What's up"}
)
conversation.predict(input="What's up with you?")
here is the response
"Ha ha, nice one, Andrew! ..."
Window Memory
We can also use a “windowed” memory where only the latest/newest conversations are remembered.
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationBufferWindowMemory(k=1)
conversation = ConversationChain(
llm=llm,
memory=memory,
verbose=True,
)
Here is our first conversation. I ask the LLM to respond me in Chinese and to be as concise as possible.
conversation.predict(
input="你好,你会说中文吗?请用中文回答,并且尽量简洁。"
)
'哈喽!我可以说中文 ...' # LLM response
Here is our second conversation.
conversation.predict(input="我的名字叫 XXX,你呢")
'哈喽 XXX!我的名称是 LLaMA ...' # LLM response
Since we set the memory window k = 1, the next conversation will not include the initial chat. The chat will not be in Chinese anymore and it won’t be concise.
conversation.predict(input="你可以教我西班牙语吗?")
"¡Hola! Of course, I'd be happy to help ..." # LLM response
Memory with a Fixed Token size
We need tiktoken and transformers to use this memory buffer. Run
pip install tiktoken transformers
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationTokenBufferMemory(
llm=llm, max_token_limit=500
)
conversation = ConversationChain(
llm=llm,
memory=memory,
verbose=True,
)
conversation.predict(
input="你好,你会说中文吗?请用中文回答,并且尽量简洁。",
)
'你好!我可以说中文 ...' # LLM response
conversation.predict(
input="你可以用中文教我西班牙语吗?"
)
'I would be happy to help you ...' # LLM response
Summary in the Memory
We can compress the memory:
- Use an LLM to write a summary for the conversation
- Use the summary as the history
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
memory = ConversationSummaryBufferMemory(
llm=llm, max_token_limit=100
)
conversation = ConversationChain(
llm=llm,
memory=memory,
verbose=True,
)
Here are the conversations.
conversation.predict(
input="你好,你会说中文吗?请用中文回答,并且尽量简洁。"
)
'你好!我可以说中文 ...' # LLM response
conversation.predict(
input="哇塞你真厉害!"
)
'哈哈,谢谢你的夸奖! ... ' # LLM response
conversation.predict(
input="你可以用中文教我西班牙语的基本语法吗?"
)
# LLM response
"""
'System: Current summary:\n\nThe human greets the AI and asks if it can speak Chinese. The AI responds by saying hello, confirming its ability to communicate in Chinese, and noting that it has learned a large amount of Chinese data, allowing for an exchange in Chinese. The human then introduces themselves as XXX and asks about the significance of their name. The AI analyzes the name\'s components, including "Yang" as an old Chinese surname and "Yu Shi" as a traditional form of Chinese poetry, suggesting that the name may express appreciation for nature or artistic pursuit.\n\nNew lines of conversation:\n\nHuman: 你可以用中文教我西班牙语的基本语法吗?\nAI:\n\n\nNew summary:\n\nThe human greets the AI and asks if it can speak Chinese. The AI responds by saying hello, confirming its ability to communicate in Chinese, and noting that it has learned a large amount of Chinese data, allowing for an exchange in Chinese. The human then introduces themselves as Yang Yu Po and asks about the significance of their name. The AI analyzes the name\'s components, including "Yang" as an old Chinese surname and "Yu Shi" as a traditional form of Chinese poetry, suggesting that the name may express appreciation for nature or artistic pursuit. The human then asks if the AI can teach them basic Spanish grammar in Chinese, and the AI is happy to help.\n\nPlease go ahead and continue the conversation! 😊'
"""
Chains in LangChain
Here we explain the concenpt of chains from langchain.
In this section, we also read a data in the file Data.csv.
import pandas as pd
from langchain.chains import (
LLMChain, SimpleSequentialChain, SequentialChain
)
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
df = pd.read_csv('Data.csv')
LLMChain
The most simple chain. The API changed in 2024. for old LangChain Versions <= 0.1, using the following format
chain = LLMChain(llm=llm, prompt=prompt)
print(chain.run(product))
The following code exhibits the newer version (0.2+).
prompt = ChatPromptTemplate.from_template(
"What is the best name to describe "
"a company that makes {product}?"
"Give me a concise and terse answer please."
)
chain = prompt | llm
product = "Queen Size Sheet Set"
print(chain.invoke(product))
"Royal Slumber Co." # LLM Response
Simple Sequential Chain
It works for the case where each individual chains have a single input and a single output.
llm = Ollama(model=LLM_MODEL)
prompt_name = ChatPromptTemplate.from_template(
"What is the best name to describe \
a company that makes {product}?"
"Give me a concise and terse answer please."
)
prompt_desc = ChatPromptTemplate.from_template(
"Write a 20 words description for the following"
"company:{company_name}"
"Give me a concise and terse answer please."
)
chain_name = LLMChain(llm=llm, prompt=prompt_name)
chain_desc = LLMChain(llm=llm, prompt=prompt_desc)
chain = SimpleSequentialChain(
chains=[chain_name, chain_desc], verbose=True
)
chain.invoke(product)['output']
# Output
"RoyalRests Inc. provides luxurious services ..." #LLM Response
Sequential Chain
Sequential Chain allow use to construct complicated workflows
flowchart LR
A(French Review) --> B[English Review]
B --> S[English Summary]
A --> L[Language]
L --> R(French Response)
S --> R
prompt_tran = ChatPromptTemplate.from_template(
"Translate the following review to english:"
"\n\n{review}"
"\nbe concise and do not add anything other than the review text."
)
prompt_summ = ChatPromptTemplate.from_template(
"Can you summarize the following review in 1 sentence:"
"\n\n{review_english}"
"\nbe concise and do not add anything other than the review text."
)
prompt_lang = ChatPromptTemplate.from_template(
"What language is the following review:\n\n{review}"
"\nbe concise and do not add anything other than the language name."
)
prompt_resp = ChatPromptTemplate.from_template(
"Write a follow up response to the following "
"summary in the specified language:"
"\n\nSummary: {summary}\n\nLanguage: {language}"
"\nbe concise and do not add anything other than the response."
)
chain_tran = LLMChain(
llm=llm, prompt=prompt_tran, output_key="review_english"
)
chain_summ = LLMChain(
llm=llm, prompt=prompt_summ, output_key="summary"
)
chain_lang = LLMChain(
llm=llm, prompt=prompt_lang, output_key="language"
)
chain_resp = LLMChain(
llm=llm, prompt=prompt_resp, output_key="response"
)
chain = SequentialChain(
chains=[chain_tran, chain_summ, chain_lang, chain_resp],
input_variables=["review"],
output_variables=["review_english", "summary", "response"],
verbose=True
)
review = df.Review[4]
outputs = chain.invoke(review)
for term, content in outputs.items():
print(f"## {term}")
print(content + "\n")
The output is too long so we omit it.
Router Chain
- Several specialist chains for specific tasks.
- A router chain to choose which specialist chain to use.
- A default chain is called when the router can not decide.
flowchart LR
I(Input) --> R(Router)
R --> P(Physics)
R --> C(Computer Science)
R --> D(default)
from langchain.chains.router import MultiPromptChain
from langchain.chains.router.llm_router import LLMRouterChain,RouterOutputParser
Here we create the chains for the router to choose from.
physics_template = """You are a very smart physics professor. \
You are great at answering questions about physics in a concise\
and easy to understand manner. \
When you don't know the answer to a question you admit\
that you don't know.
Try to be as concise as possible.
Here is a question:
{input}"""
computerscience_template = """ You are a successful computer scientist.\
You have a passion for creativity, collaboration,\
forward-thinking, confidence, strong problem-solving capabilities,\
understanding of theories and algorithms, and excellent communication \
skills. You are great at answering coding questions. \
You are so good because you know how to solve a problem by \
describing the solution in imperative steps \
that a machine can easily interpret and you know how to \
choose a solution that has a good balance between \
time complexity and space complexity.
Try to be as concise as possible.
Here is a question:
{input}"""
prompts = [
{
"name": "physics",
"description": "Good for answering questions about physics",
"prompt_template": physics_template
},
{
"name": "computer science",
"description": "Good for answering computer science questions",
"prompt_template": computerscience_template
}
]
destination_chains = {}
for p_info in prompts:
name = p_info["name"]
prompt_template = p_info["prompt_template"]
prompt = ChatPromptTemplate.from_template(template=prompt_template)
chain = LLMChain(llm=llm, prompt=prompt)
destination_chains[name] = chain
default_prompt = ChatPromptTemplate.from_template("{input}")
default_chain = LLMChain(llm=llm, prompt=default_prompt)
Here we prepare for the router chain
destinations = [f"{p['name']}: {p['description']}" for p in prompts]
destinations_str = "\n".join(destinations)
print(destinations_str)
# Output
"""
physics: Good for answering questions about physics
computer science: Good for answering computer science questions
"""
Now we use this very long prompt to guild the LLM.
MULTI_PROMPT_ROUTER_TEMPLATE = r"""Given a raw text input to a \
language model select the model prompt best suited for the input. \
You will be given the names of the available prompts and a \
description of what the prompt is best suited for. \
You may also revise the original input if you think that revising\
it will ultimately lead to a better response from the language model.
<< FORMATTING >>
Return a markdown code snippet with a JSON object formatted to look like:
\`\`\`json
{
"destination": string \ name of the prompt to use or "DEFAULT"
"next_inputs": string \ a potentially modified version of the original input
}
\`\`\`
REMEMBER: "destination" MUST be one of the candidate prompt \
names specified below OR it can be "DEFAULT" if the input is not\
well suited for any of the candidate prompts.
REMEMBER: "next_inputs" can just be the original input \
if you don't think any modifications are needed.
<< CANDIDATE PROMPTS >>
{destinations}
<< INPUT >>
<< OUTPUT (remember to include the ```json)>>
"""
router_template = MULTI_PROMPT_ROUTER_TEMPLATE.format(
destinations=destinations_str
)
router_prompt = PromptTemplate(
template=router_template,
input_variables=["input"],
output_parser=RouterOutputParser(),
)
Now we create the chain with the router.
router_chain = LLMRouterChain.from_llm(llm, router_prompt)
chain = MultiPromptChain(
router_chain=router_chain,
destination_chains=destination_chains,
default_chain=default_chain, verbose=True
)
# LLM will choose Physics
chain.invoke("What is Percus Yevick equation?")
# LLM will use Computer Science
chain.invoke("what is the travelling salesman problem")
# Console Output
"""
→ Entering new MultiPromptChain chain...
✦ computer science: {
'input': "Is there a specific solution ...?"
}
→ Finished chain.
"""
# LLM response
"""
A classic problem!
For TSP, I'd recommend exploring dynamic programming
approaches like Christofides Algorithm or 2-Opt
Heuristics. These methods balance time and space
complexity well.
"""
# LLM will use Computer Science
chain.invoke("what is NGS?")
# Console Output
"""
→ Entering new MultiPromptChain chain ...
computer science: {
'input': 'What is Next Generation Sequencing (NGS)?'
}
→ Finished chain.
"""
# LLM Response
"""
Next-Generation Sequencing (NGS) refers to the technology
used in DNA sequencing, which allows for rapid and
cost-effective analysis of an individual's entire genome
or specific regions of interest. NGS enables the simultaneous
analysis of millions of DNA fragments, resulting in faster
turnaround times and higher resolution than traditional
Sanger sequencing methods.
"""
QA for Documents
from langchain.document_loaders import CSVLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.vectorstores import DocArrayInMemorySearch
from langchain_community.embeddings import OllamaEmbeddings
from langchain.chains import RetrievalQA
from IPython.display import display, Markdown
Overview
To do a RAG we need
- an embedding model
- a vector database
- chunk the documents and index them in the vector DB
- match query with the embedding model
- generate the response with the LLM
Creating the Vector DB
EMBEDDING_MODEL = "all-minilm"
file = 'OutdoorClothingCatalog_1000.csv'
loader = CSVLoader(file_path=file)
index = VectorstoreIndexCreator(
embedding=OllamaEmbeddings(model=EMBEDDING_MODEL),
vectorstore_cls=DocArrayInMemorySearch
).from_loaders([loader])
Generate response given a query
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
query ="""\
Please list all your shirts with sun protection
in a table in markdown and summarize each one.
"""
response = index.query(query, llm=llm)
display(Markdown(response))
Here are the shirts with sun protection listed in a table:
| Shirts | Description | Sun Protection |
|---|---|---|
| Sun Shield Shirt | High-performance sun shirt for UV ray protection | SPF 50+ (blocks 98% of sun’s harmful rays) |
| Tropical Breeze Shirt | Lightweight, breathable long-sleeve UPF shirt for superior sun protection | SPF 50+ (blocks 98% of sun’s harmful rays) |
| Women’s Tropical Tee, Sleeveless | Five-star sleeveless button-up shirt with SunSmart UPF 50+ rating | SPF 50+ (blocks 98% of sun’s harmful rays) |
| Girls’ Beachside Breeze Shirt, Half-Sleeve | Rash guard-style swim shirt with built-in UPF 50+ protection | SPF 50+ (blocks 98% of sun’s harmful rays) |
Summary: All the shirts listed have high-performance fabric with SPF 50+ sun protection, blocking 98% of the sun’s harmful UV rays. They are suitable for outdoor activities such as swimming, fishing, and travel to protect skin from damage.
RAG Step-by-Step
Load the Documents
loader = CSVLoader(file_path=file)
docs = loader.load()
# docs[0] print
"""
Document(
metadata={
'source': 'OutdoorClothingCatalog_1000.csv', 'row': 0
},
page_content="..."
)
"""
Create the Vector Database
embeddings = OllamaEmbeddings(model=EMBEDDING_MODEL)
embed = embeddings.embed_query("Hi my name is Yushi")
len(embed) # -> 384
embed[:5] # [-0.4055, 0.2655, 0.1268, 0.4260, -0.5726]
# vector database in memory
db = DocArrayInMemorySearch.from_documents(
docs, embeddings
)
Generate the Response
llm = Ollama(model=LLM_MODEL, verbose=True)
retriever = db.as_retriever()
query = "Please suggest a shirt with sunblocking"
docs = db.similarity_search(query, k=4)
qdocs = "".join(
[docs[i].page_content for i in range(len(docs))]
)
response = llm.invoke(
f"{qdocs} Question: Please list all your \
shirts with sun protection in a table in markdown"
)
display(Markdown(response))
Here is the list of shirts with sun protection in a table format using Markdown:
| Shirt Name | Description | Sun Protection |
|---|---|---|
| Sun Shield Shirt | Block the sun, not the fun - our high-performance sun shirt is guaranteed to protect from harmful UV rays. | UPF 50+ rated, blocks 98% of the sun’s harmful rays |
| Tropical Breeze Shirt | Beat the heat in this lightweight, breathable long-sleeve men’s UPF shirt, offering superior SunSmart protection from the sun’s harmful rays. | UPF 50+ rated, blocks 98% of the sun’s harmful rays |
| Women’s Tropical Tee, Sleeveless | Our five-star sleeveless button-up shirt has a fit to flatter and SunSmart protection to block the sun’s harmful UV rays. | UPF 50+ rated, blocks 98% of the sun’s harmful rays |
| Men’s Plaid Tropic Shirt, Short-Sleeve | Our Ultracomfortable sun protection is rated to UPF 50+, helping you stay cool and dry. | UPF 50+ rated, blocks 98% of the sun’s harmful rays |
Note: All shirts have a UPF (Ultraviolet Protection Factor) rating of 50+, which means they block at least 95% of UVB rays and at least 90% of UVA rays.
Retrieval QA Chain
The RetrievalQA chain can achieve identical RAG results.
qa_stuff = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff", # "stuffs" all documents into context
retriever=db.as_retriever(),
verbose=True
)
query = "Please list all your shirts with sun protection"
query += " in a table in markdown."
response = qa_stuff.invoke(query)
# console output
"""
→ Entering new RetrievalQA chain
→ Finished chain.
"""
display(Markdown(response['result']))
Here are the shirts with sun protection listed in a table:
| Shirt Name | Sun Protection |
|---|---|
| Sun Shield Shirt by [name] | UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays) |
| Women’s Tropical Tee, Sleeveless | UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays) |
| Tropical Breeze Shirt | UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays) |
| Girls’ Beachside Breeze Shirt, Half-Sleeve | UPF 50+, SPF 50+ (blocks 98% of sun’s harmful rays) |
Let me know if you need anything else!
Retrievel Methods
Stuff
put all retrieved chunks into the context. Then call LLM once. The most common method.
flowchart LR
A[Documents] --> B(chunk)
A --> C(chunk)
A --> D(chunk)
B --> L[LLM]
C --> L
D --> L --> R[Response]
Map Reduce
pass each chunk and query to LLM, then use LLM to process the responses. The second most common method.
flowchart LR
A[Documents] --> B(chunk) --> LB[LLM]
A --> C(chunk) --> LC[LLM]
A --> D(chunk) --> LD[LLM]
LB --> L[LLM]
LC --> L
LD --> L --> R[Response]
Refine
Builds the answer iteratively.
flowchart LR
A[Documents] --> B(chunk) --> LB[LLM]
A --> C(chunk) --> LC[LLM]
A --> D(chunk) --> LD[LLM]
LB --> LC
LC --> LD
LD --> R[Response]
Map Rerank
flowchart LR
A[Documents] --> B(chunk) --> LB[LLM] --> SB(Score 40)
A --> C(chunk) --> LC[LLM] --> SC(Score 60) --> R[Response]
A --> D(chunk) --> LD[LLM] --> SD(Score 32)
LLM Evaluation
General Instruction
- Visualisation: to understand the input/output data.
- Create examples for evaluation.
- Use LLM to evaluate.
Here is a Step by Step guide.
Setting Up
file = 'OutdoorClothingCatalog_1000.csv'
loader = CSVLoader(file_path=file)
data = loader.load()
index = VectorstoreIndexCreator(
embedding=OllamaEmbeddings(model=EMBEDDING_MODEL),
vectorstore_cls=DocArrayInMemorySearch
).from_loaders([loader])
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=index.vectorstore.as_retriever(),
)
To evaluate the result, we should create standard “Question” + “Answer” examples.
Creating Examples Manually
We can generate such examples (Question + Answer) by
- Observing the data.
- Asking a question ourselves.
- Answer the question ourselves.
Here is an example
# data[10].page_content
: 10
name: Cozy Comfort Pullover Set, Stripe
description: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.
Size & Fit
- Pants are Favorite Fit: Sits lower on the waist.
- Relaxed Fit: Our most generous fit sits farthest from the body.
Fabric & Care
- In the softest blend of 63% polyester, 35% rayon and 2% spandex.
Additional Features
- Relaxed fit top with raglan sleeves and rounded hem.
- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.
Imported.
Here is the manually crafted example.
examples = [
{
"query": "Do the Cozy Comfort Pullover Set have side pockets?",
"answer": "Yes"
},
{
"query": "What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?",
"answer": "The DownTek collection"
}
]
Generating Examples with LLMs
the following code was copied from
langchain/evaluation/qa/generate_chain.pySo that our code will be compatible with the new interface for langchain 0.2+
from langchain.output_parsers.regex import RegexParser
qa_gen_parser = RegexParser(
regex=r"QUESTION: (.*?)\n+ANSWER: (.*)", output_keys=["query", "answer"]
)
qa_gen_template = """\
You are a teacher coming up with questions to ask on a quiz.\
Given the following document, please generate a question and\
answer based on that document.
Example Format:
<Begin Document>
...
<End Document>
QUESTION: question here
ANSWER: answer here
These questions should be detailed and be based explicitly on\
information in the document. Begin!
<Begin Document>
{doc}
<End Document>"""
qa_gen_prompt = PromptTemplate(
input_variables=["doc"],
template=qa_gen_template,
)
We now build the QAGenerateChain with the runable interface.
For old LangChain version (<= 0.1), we can use the following code
qa_gen = QAGenerateChain.from_llm(llm) new_examples = qa_gen.apply_and_parse( [{"doc": t} for t in data[:5]] )
LLM_MODEL = "llama3"
llm = Ollama(model=LLM_MODEL)
qa_gen = qa_gen_prompt | llm | qa_gen_parser
new_examples = qa_gen.batch(
[{"doc": t} for t in data[:5]]
)
# new examples
[{
'query': "What type of material is ... ?",
'answer': 'Soft canvas material.'
}, {
'query': 'What percentage of recycled ... ?',
'answer': '94%'
}, {
'query': "What feature of this toddler ... ?",
'answer': 'The UPF 50+ rated fabric.'
}, {
'query': "What percentage of the ... ?",
'answer': '82%'
}, {
'query': 'What technology does EcoFlex ... ?',
'answer': 'TEK O2 technology.'
}]
Manually Evaluate
examples += new_examples
qa.invoke(examples[0]["query"])
# output
{
'query': 'Do the Cozy Comfort Pullover Set have side pockets?',
'result': 'According to ... the answer is:\n\nYes ...'
}
examples[0]["answer"] # Output: 'Yes'
LLM assisted evaluation
The default prompt can be found in the file
langchain/evaluation/qa/eval_prompt.py
Here we edited the prompt so that the output is good for llama3.
from langchain.evaluation.qa import QAEvalChain
grade_template = """You are a teacher grading a quiz.
You are given a question, the student's answer, and the true answer,
and are asked to score the student answer as either CORRECT or INCORRECT.
Example Input Format:
-----
QUESTION: question here
STUDENT ANSWER: student's answer here
TRUE ANSWER: true answer here
-----
Example Output:
CORRECT or INCORRECT
Grade the student answers based ONLY on their factual accuracy.
Ignore differences in punctuation and phrasing between the student answer and true answer.
It is OK if the student answer contains more information than the true answer,
as long as it does not contain any conflicting statements.
Return the final grade. The only options are between CORRECT and INCORRECT.
Here are the actual input data:
QUESTION: {query}
STUDENT ANSWER: {result}
TRUE ANSWER: {answer}
"""
grade_prompt = PromptTemplate(
input_variables=["query", "result", "answer"],
template=grade_template
)
eval_chain = QAEvalChain.from_llm(llm, prompt=grade_prompt)
predictions = qa.batch(examples)
graded_outputs = eval_chain.evaluate(examples, predictions)
# graded_outputs content
[{'results': 'CORRECT'},
{'results': 'CORRECT'},
{'results': 'CORRECT'},
{'results': 'CORRECT'},
{'results': 'CORRECT'},
{'results': 'CORRECT'},
{'results': 'CORRECT'}]
for i, eg in enumerate(examples):
print(f"Example {i}:")
print("Question: " + predictions[i]['query'])
print("Real Answer: " + predictions[i]['answer'])
print("Predicted Answer: " + predictions[i]['result'])
print("Predicted Grade: " + graded_outputs[i]['results'])
print()
# concole output
"""
Example 0:
Question: Do the Cozy Comfort Pullover Set have side pockets?
Real Answer: Yes
Predicted Answer: According to the ...
Predicted Grade: CORRECT
Example 1:
Question: What collection is ... ?
Real Answer: The DownTek collection
Predicted Answer: The Ultra-Loft 850 Stretch ...
Predicted Grade: CORRECT
Example 2:
Question: What type of material is ... ?
Real Answer: Soft canvas material.
Predicted Answer: According to the description ...
Predicted Grade: CORRECT
Example 3:
Question: What percentage of recycled ... ?
Real Answer: 94%
Predicted Answer: According to the context ... the answer is 94%.
Predicted Grade: CORRECT
Example 4:
Question: What feature of this toddler's swimsuit ... ?
Real Answer: The UPF 50+ rated fabric.
Predicted Answer: Based on the context provided, the answer ...
Predicted Grade: CORRECT
Example 5:
Question: What percentage of the swimtop's body ...?
Real Answer: 82%
Predicted Answer: Based on the context, I found ...
Predicted Grade: CORRECT
Example 6:
Question: What technology does EcoFlex 3L ... ?
Real Answer: TEK O2 technology.
Predicted Answer: According to the context, EcoFlex ...
Predicted Grade: CORRECT
"""
Debug
We can use the following setup to use the debug mode and see the exact input/ouput for the LLM.
from langchain.globals import set_debug
set_debug(True)
response = qa.invoke(examples[0]["query"])
# Massive amount of information will be printed.
set_debug(False) # turn off debug mode
Agents
- set temperature to zero to get precise results.
- we use the
OllamaFunctionsto support local LLM.
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain.agents import load_tools, create_react_agent, AgentExecutor
from langchain_core.pydantic_v1 import BaseModel, Field
Creating a tool calling LLM
Detail: Ollama model
We have to install langchain-experimental to use the ollama model.
pip install langchain_experimental
I also changed the default system prompt a bit so that llama3 is more likely to return a json file.
Code
LLM_MODEL = "llama3"
TOOL_PROMPT = """
You have access to the following tools:
{tools}
You must always select one of the above tools and respond with only a
JSON object matching the following schema:
{
"tool": <name of the selected tool>,
"tool_input": <parameters for the selected tool, matching the tool's JSON schema>
}
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
"""
llm = OllamaFunctions(
model=LLM_MODEL,
temperature=0.0,
verbose=True,
tool_system_prompt_template=TOOL_PROMPT
)
class GetWeather(BaseModel):
"""Get the current weather in a given location"""
location: str = Field(
..., description="The city and state, e.g. San Francisco, CA"
)
llm_with_tools = llm.bind_tools([GetWeather])
ai_message = llm_with_tools.invoke(
"what is the weather like in San Francisco",
)
# ai_message.tool_calls
[{
'name': 'GetWeather',
'args': {'location': 'San Francisco, CA'},
'id': 'call_4eb652ffba9547b48ab49cc301d50385'
}]
"""
Creating an Agent
For old langchain version, the agent can be created via the following code.
from langchain.agents import initialize_agent, AgentType tools = load_tools(["llm-math","wikipedia"], llm=llm) agent= initialize_agent( tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, handle_parsing_errors=True, verbose=True )
Here we use the newer interface for langchain 0.2+.
LLM_MODEL = "llama3"
TOOL_PROMPT = """
You have access to the following tools:
{tools}
You must always select one of the above tools and respond with\
only a JSON object matching the following schema:
{
"tool": <name of the selected tool>,
"tool_input": <parameters for the selected tool, matching the tool's JSON schema>
}
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
DO NOT RESPOND ANYTHING RATHER THAN THE JSON CONTENT !!!
"""
llm = OllamaFunctions(
model=LLM_MODEL,
temperature=0.0,
verbose=True,
tool_system_prompt_template=TOOL_PROMPT
)
react_template = '''
Answer the following questions as best you can.
You have access to the following tools:
{tools}
Use the following format:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
Follow the follwoing rules strictly no matter what
- DO NOT ADD EXTRA DESCRIPTION FOR the action and action input!
- DO NOT REPEAT ANY INFORMATION!
- DO NOT REPEAT THE QUESTION!
- FOLLOW THE FORMAT REQUIREMENT!
Begin!
Question: {input}
Thought:{agent_scratchpad}
'''
react_prompt = PromptTemplate.from_template(react_template)
tools = load_tools(["wikipedia", "wikipedia"], llm=llm)
agent= create_react_agent(llm, tools, prompt=react_prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
We can interact with the agent via the invoke method.
question = "Who is Joe Biden"
result = agent_executor.invoke({"input": question})
# console output
"""
⇨ Entering new AgentExecutor chain...
⇨ Here's my response:
Thought: I need to find information about Joe Biden.
Action: wikipedia
Action Input: Joe Biden
▶ Page: Joe Biden
Summary: Joseph Robinette Biden ...
▶ Page: Family of Joe Biden
Summary: Joe Biden, the 46th and ...
▶ Page: Presidency of Joe Biden
summary: Joe Biden's tenure as the 46th president ...
Final Answer: Joe Biden is an American politician ...
⇨ Finished chain.
"""
Custom Tool
from langchain.agents import tool
from datetime import date
we create a new tool called timer.
@tool
def timer(text: str) -> str:
"""Returns todays date, use this for any \
questions related to knowing todays date. \
The input should always be an empty string, \
and this function will always return todays \
date - any date mathmatics should occur \
outside this function."""
return str(date.today())
We use the tool like this
tools = load_tools(["wikipedia"], llm=llm)
tools.append(timer)
agent= create_react_agent(llm, tools, prompt=react_prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
question = "what is the time now?"
result = agent_executor.invoke(
{"input": question}
)
# console output
"""
⇨ Entering new AgentExecutor chain...
Here's my attempt at answering your question:
Thought:
Action: timer
Action Input: 2024-07-19 Here's the answer:
Thought: I now know the final answer
Final Answer: 2024-07-19
⇨ Finished chain.
"""