Di's Blog

Persistence in LangGraph

Persistence is a cornerstone for building robust and production-grade applications. LandGraph introduces a game-changing feature that ensures application states are stored and retrievable at any point. This redefines reliability and scalability in workflow management. This capability is especially vital when executing workflows involving interruptions, user inputs, or debugging. Whether you're building a simple app or an enterprise-grade system, persistence ensures your application is always ready to handle interruptions and user interactions gracefully.

The "Persisting Agent Stage" enables seamless workflows, especially in user-facing applications. Here’s why this feature is critical:

1. Human-in-the-Loop Workflows: Many applications rely on user input to make decisions or advance processes. With persistence, LandGraph allows the graph execution to pause, checkpoint the state into persistent storage, and resume later. This means the application can wait for user input and continue without losing context.
2. Debugging and History: Persistence creates a robust mechanism for saving the application state after every step. This makes debugging easier and enables the creation of detailed execution histories.
3. Support for Multi-Session Scenarios: Applications often require users to switch between sessions while maintaining their progress. Persistence ensures continuity by saving states into persistent storage.

At the heart of this feature is the CheckPointer object, a persistence layer implemented by LandGraph. Here’s how it works:

• Integration with Databases The CheckPointer can save states into various database types, including:

  • Document databases: Firestore, MongoDB
  • Relational databases: PostgreSQL, SQLite, MySQL
  • Graph databases: Neo4j, AWS Neptune

  For example, the following section will focus on persisting states into an SQLite database, a popular choice for local environments. The process can also be extended to managed cloud databases like Google Cloud SQL or AWS RDS.

• State Management As each node in the graph executes, the CheckPointer saves the updated state into the database. This ensures that states are recoverable after interruptions, enabling the graph to resume execution from exactly where it left off.

To implement persistence, follow these simple steps:

1. Import the CheckPointer object from LandGraph.
2. Create an instance of CheckPointer and configure it with a connection string (local or cloud-based database).
3. Pass the CheckPointer instance to your graph during creation. LandGraph will handle state persistence automatically after each node execution.

1
2
3
4

from langgraph.checkpoint.sqlite import SqliteSaver

memory = SqliteSaver.from_conn_string(":checkpoints.sqlite:")
graph = workflow.complie(checkpointer=memory)

The result is that you can pause the graph, fetch user input, and continue execution seamlessly, all while ensuring states are securely stored in your chosen database.

MemorySaver + Interrupts = Human In The Loop

Human-in-the-loop systems are essential to modern applications, allowing seamless integration of human feedback into automated workflows. With the help of the MemorySaver feature, you can build applications using LangGraph that pause, capture user input, and resume execution effortlessly.

In workflows involving human interaction, there are moments where the application needs to pause, gather feedback from the user, and then continue processing. For instance, consider a sequence of tasks where:

1. A process executes its initial steps.
2. The system pauses to collect human input.
3. The workflow resumes, incorporating the user’s feedback.

This type of flow requires interrupts to halt the execution and persistence to save the current state of the workflow. Langraph provides the tools to manage both seamlessly.

Implementation

To illustrate, let’s build a straightforward graph with the following steps:

1. Start with a simple initial node.
2. Execute a task and pause for human feedback.
3. Resume execution with the updated state and complete the workflow.

We use Langraph's MemorySaver, a checkpointing tool that saves the workflow’s state in memory after each node’s execution. This ephemeral storage method is perfect for local testing and prototyping. Here’s a simplified version of the setup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

from dotenv import load_dotenv

load_dotenv()
from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver


class State(TypedDict):
    input: str
    user_feedback: str


def step_1(state: State) -> None:
    print("---Step 1---")


def human_feedback(state: State) -> None:
    print("---human_feedback---")


def step_3(state: State) -> None:
    print("---Step 3--")


builder = StateGraph(State)
builder.add_node("step_1", step_1)
builder.add_node("human_feedback", human_feedback)
builder.add_node("step_3", step_3)
builder.add_edge(START, "step_1")
builder.add_edge("step_1", "human_feedback")
builder.add_edge("human_feedback", "step_3")
builder.add_edge("step_3", END)


memory = MemorySaver()

graph = builder.compile(checkpointer=memory, interrupt_before=["human_feedback"])

graph.get_graph().draw_mermaid_png(output_file_path="graph.png")

The graph visualization by using Mermaid.ink is here:

MemorySaver Implementations

Integrating human feedback into automated systems is a growing trend in AI development. It bridges the gap between machine automation and human judgment, enabling better decision-making, improved accuracy, and adaptability. In this section, we explore how to incorporate human-in-the-loop functionality into a graph-based system while leveraging memory storage to track execution states. This walkthrough showcases the process from initialization to final execution.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

from dotenv import load_dotenv

load_dotenv()
from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver


class State(TypedDict):
    input: str
    user_feedback: str


def step_1(state: State) -> None:
    print("### Step 1 ###")


def human_feedback(state: State) -> None:
    print("### Human Feedback ###")


def step_3(state: State) -> None:
    print("### Step 3 ###")


builder = StateGraph(State)
builder.add_node("step_1", step_1)
builder.add_node("human_feedback", human_feedback)
builder.add_node("step_3", step_3)
builder.add_edge(START, "step_1")
builder.add_edge("step_1", "human_feedback")
builder.add_edge("human_feedback", "step_3")
builder.add_edge("step_3", END)


memory = MemorySaver()

graph = builder.compile(checkpointer=memory, interrupt_before=["human_feedback"])

graph.get_graph().draw_mermaid_png(output_file_path="graph.png")

if __name__ == "__main__":
    thread = {"configurable": {"thread_id": "1"}}

    initial_input = {"input": "hello world"}

    for event in graph.stream(initial_input, thread, stream_mode="values"):
        print(event)

    print(graph.get_state(thread).next)

    user_input = input("How do you want to update the state? ")

    graph.update_state(thread, {"user_feedback": user_input}, as_node="human_feedback")

    print("### State after update ###")
    print(graph.get_state(thread))

    print(graph.get_state(thread).next)

    for event in graph.stream(None, thread, stream_mode="values"):
        print(event)

The graph’s execution is tied to a thread variable, a dictionary initialized with a thread_id. This serves as a session or conversation identifier, distinguishing various graph runs. For simplicity, the thread_id is set to 1, though a more robust implementation would use a UUID. The graph processes events using graph.stream(), which accepts the initial input and thread details. Events are streamed in value mode, and each event is printed for transparency.

During execution:

• Input is processed.
• Node executions are logged.
• Interruptions allow for dynamic human input.

Running the graph in debug mode provides insights into:

• Memory storage (memory.storage) containing nested objects that log the graph state.
• Transition logs for each node, showing updates or lack thereof.

At an interrupt, human feedback is solicited using Python's built-in input() function. This input updates the state dynamically. Once human input is integrated, the graph resumes execution. Subsequent steps process the updated state, leading to the graph’s completion.

SqliteSaver

Switching from an ephemeral memory-based state saver to a persistent database saver can significantly enhance the durability and traceability of your graph’s execution. In this section, we’ll explore how to replace the in-memory MemorySaver with an SQLiteSaver for long-term storage and easy debugging.

The MemorySaver is transient, meaning all state information vanishes after the program stops. By using an SQLite database, you can:

• Persist graph states across runs.
• Debug and troubleshoot using a structured database.
• Resume executions exactly where they were interrupted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

import sqlite3

from dotenv import load_dotenv
from langgraph.checkpoint.sqlite import SqliteSaver

load_dotenv()
from typing import TypedDict
from langgraph.graph import StateGraph, START, END


class State(TypedDict):
    input: str
    user_feedback: str


def step_1(state: State) -> None:
    print("### Step 1 ###")


def human_feedback(state: State) -> None:
    print("### Human Feedback ###")


def step_3(state: State) -> None:
    print("### Step 3 ###")


builder = StateGraph(State)
builder.add_node("step_1", step_1)
builder.add_node("human_feedback", human_feedback)
builder.add_node("step_3", step_3)
builder.add_edge(START, "step_1")
builder.add_edge("step_1", "human_feedback")
builder.add_edge("human_feedback", "step_3")
builder.add_edge("step_3", END)

conn = sqlite3.connect("checkpoints.sqlite", check_same_thread=False)
memory = SqliteSaver(conn)
graph = builder.compile(checkpointer=memory, interrupt_before=["human_feedback"])

graph.get_graph().draw_mermaid_png(output_file_path="graph.png")

if __name__ == "__main__":
    thread = {"configurable": {"thread_id": "1"}}

    initial_input = {"input": "hello world"}

    for event in graph.stream(initial_input, thread, stream_mode="values"):
        print(event)

    print(graph.get_state(thread).next)

    user_input = input("How do you want to update the state: ")

    graph.update_state(thread, {"user_feedback": user_input}, as_node="human_feedback")

    print("### State after update ###")
    print(graph.get_state(thread))

    print(graph.get_state(thread).next)

    for event in graph.stream(None, thread, stream_mode="values"):
        print(event)

We start by importing the required modules. Then Initialize a connection to your SQLite database. The check_same_thread=False flag ensures thread-safe database operations, essential for stopping and restarting execution across different threads. After that we create an instance of SQLiteSaver and pass it the SQLite connection. This saver integrates seamlessly with the graph execution pipeline, persisting states to the SQLite database.

1. Initial Execution: Run the graph with the SQLiteSaver. After execution, you’ll see a new file, checkpoints.sqlite, created in your project directory.
2. Inspect the Database: Use your IDE’s database tools (e.g. SQLite3 Editor for VS Code) to load and inspect the checkpoints.sqlite file. You’ll find a table storing graph states, similar to what you’d see with MemorySaver, but now it’s persistent.

Changing the thread_id allows you to simulate a new session while retaining access to previous runs. When resuming, the graph starts from the last recorded state. You can verify this by inspecting the database entries for the new thread_id.

For enhanced traceability, integrate Langsmith for tracking and debugging. Langsmith provides detailed insights, including thread metadata and execution traces.

Substack

How Meta Plans To Crank the Dopamine Machine With Infinite AI-Generated Content - The Algorithmic Bridge [Link]

This article discussed AI’s most dangerous potential - its ability to manipulate and addict humans through hyper-targeted entertainment. This trend, spearheaded by companies like Meta, risks reshaping human cognition and agency, raising existential questions about freedom, pleasure, and the future of society.

One good point is that the killer robots are brain-hacking entertainment. A very plausible dystopia involves technology (e.g., AI-driven entertainment) manipulating human attention and cognition for profit. Traditional TV was a prototype of mental manipulation but lacked personalization. Current platforms such as Netflix and TikTok use algorithms to cater to preferences but still feel limited. Future AI will create hyper-personalized content tailored to individual preferences in real-time, exploiting human psychology. Meta’s generative AI plans are the next step toward addictive, manipulative entertainment. Meta announced that AI content creators will be designed to enhance engagement on platforms like Facebook and Instagram. Connor Hayes, Meta’s VP for generative AI, explained how AI accounts will create and share engaging content.

The Five Stages of AGI Grief - Marcus on AI [Link]

Marcus uses the framework of the Kübler-Ross model of grief to describe the emotional responses people are having (or will likely have) to the ongoing developments in Artificial General Intelligence (AGI). He argues that many people are not yet facing the reality of AGI and are likely to go through similar stages of grief as it gets closer.

1. Denial: Many people, including some experts, are still in denial about the possibility and speed of AGI development. They dismiss the progress, underestimate its potential, or claim it's decades away.
2. Anger: Once denial fades, anger emerges, often directed at those perceived as enabling or hyping AGI. This can be targeted at AI researchers, tech companies, or even the technology itself.
3. Bargaining: In this stage, people try to find ways to control or mitigate AGI, often through unrealistic expectations or proposed solutions.
4. Depression: As bargaining fails, a sense of profound unease and hopelessness may set in. This is the realization that AGI could fundamentally change society in ways that are difficult to predict or control, leading to feelings of powerlessness.
5. Acceptance: This is the final stage, where people begin to accept the reality of AGI and its potential impact. This isn't necessarily cheerful, but it's characterized by a shift from denial and fear towards a more realistic view.

The Taiwan posts - Noahpinion [Link]

Disney Paid Off Trump for a Reason - BIG by Matt Stoller [Link]

Fubo, a sports streaming service, had previously won a preliminary injunction against a joint venture between Disney, Fox, and Warner Bros, arguing that the venture was an illegal merger. However, Fubo's stock wasn't performing well, leading Fubo CEO David Gandler to sell a controlling stake in his company to Disney.

Here are the rationales behind this decision, according to the sources:

• Fubo's CEO, David Gandler, profited from winning an antitrust suit and joined forces with a large corporation. Instead of being an underdog fighting against major corporations, Fubo has now joined forces with one of them. Fubo will now have Disney's resources, while its leaders imagine that it will operate somewhat independently.
• Disney made a $16 million payment for defamation against Trump, which is considered questionable by legal analysts, in order to gain credibility with Trump. The aim of this was to ensure that government enforcers would not interfere with the deal.
• Fubo's leaders may be ignoring the risks involved in the merger. They are potentially exhibiting a kind of "malevolent naivete" and airbrushing away their own violation of the law.

The sources suggest that Fubo's leadership may not be considering some of the risks associated with mergers. Mergers carry significant risk, and they can fall apart for a variety of reasons. During the 18-24 months that it takes to clear financing and regulatory hurdles, a company under contract to be sold cannot make significant strategic decisions or investments, while the purchaser can do whatever they want. If the deal falls apart, the company that was to be sold could be in a significantly worse position.

The sources point out that there is a possibility that another private litigant could take Fubo's place and sue, using the legal precedent set by Fubo. This is evidenced by a letter sent by EchoStar to the court, in which the company states that it's considering suing along the same lines as Fubo. This may not matter to Disney, since they now control Fubo, but it should be a source of concern for Fubo's leadership team who have essentially bet their company on a violation of the law.

A private litigant, such as EchoStar, could take Fubo's place and sue Disney, Fox, and Warner Bros, using the same legal arguments that Fubo successfully used to win a preliminary injunction. This is a possibility because the legal precedent set by Fubo remains, even though Fubo is now under Disney's control.

Here's why this could be problematic for Fubo but not necessarily for Disney:

• Fubo is in a vulnerable position due to the merger agreement. While the deal is pending, Fubo is restricted in its strategic decision-making and investments, effectively putting the company in "limbo". This means Fubo cannot make significant moves to respond to a new lawsuit.
• Disney, as the purchaser, is not similarly restricted. They can continue to operate as they see fit. They have the resources to handle a new legal challenge.
• If the merger fails, Fubo will have wasted 18-24 months with the potential for no significant strategic moves. It could end up in a weakened state compared to competitors who were not in a merger process. The company might even become "a limping and probably dead company". Failed mergers can also lead to leadership changes, such as the CEO getting fired.
• Disney has already taken steps to ensure the deal's success, including a payment to gain credibility with the current administration. While another lawsuit could present a challenge, Disney has the resources and political connections to navigate it, which Fubo does not.
• The incentive to complete the deal is different for Disney and Fubo. Disney will remain a major player regardless of the deal's outcome. However, Fubo's future is heavily dependent on the merger. This makes Fubo more vulnerable if the deal is challenged.

The rise and fall of "fact-checking" - Silver Bulletin [Link]

The main opinion of this article is that Meta's decision to replace fact-checkers with a community notes system is justifiable because fact-checkers have been politically biased and have not effectively addressed the issue of misinformation.

While the author agrees with Zuckerberg's decision, they also acknowledge that Zuckerberg's motivations may not be high-minded, but rather driven by political pressure and business incentives. Despite that, the author thinks the move is "pointing in the right direction," and agrees with Zuckerberg's claim that fact-checkers have been too politically biased. The author also admits their own biases and that Community Notes is a new program that might also have problems.

US Banks: Profits Surge - App Economy Insights [Link]

CES 2025: AI Takes Over - App Economy Insights [Link]

a16z's big ideas in tech for 2025 - ben lang's notes [Link]

Andreessen Horowitz’s list of big ideas in tech for 2025:

How AI-assisted coding will change software engineering: hard truths - The Pragmatic Engineer [Link]

Great article!

This "70% problem" suggests that current AI coding tools are best viewed as:

• Prototyping accelerators for experienced developers
• Learning aids for those committed to understanding development
• MVP generators for validating ideas quickly

Current tools mostly wait for our commands. But look at newer features like Anthropic's computer use in Claude, or Cline's ability to automatically launch browsers and run tests. These aren't just glorified autocomplete - they're actually understanding tasks and taking initiative to solve problems.

Think about debugging: Instead of just suggesting fixes, these agents can:

• Proactively identify potential issues
• Launch and run test suites
• Inspect UI elements and capture screenshots
• Propose and implement fixes
• Validate the solutions work (this could be a big deal)

― The 70% problem: Hard truths about AI-assisted coding - Elevate [Link]

Great pragmatic article! And it's well-said in the end: "Software quality was (perhaps) never primarily limited by coding speed...The goal isn't to write more code faster. It's to build better software. "

AI tools help experienced developers more than beginners. This is similar to the fact that AI helps top biologists to be successful more than normal biologists. The results and efficiency of AI usage differ based on users' domain expertise. This is called 'knowledge paradox'. AI can help to get the first 70% job done quickly, but the efforts on the final 30% have diminishing returns. This is called 'AI learning curve paradox'.

o1 isn’t a chat model (and that’s the point) - Latent Space [Link]

• Provide Extensive Context: Give 10x more context than you think is necessary. This includes details about previous attempts, database schemas, and company-specific information. Think of o1 like a new hire that needs all the relevant information to understand the task. Put the context at the end of your prompt.

  Use tools like voice memos to capture context and paste transcripts. You can also save reusable segments of context for future use. AI assistants within other products can help extract context.

• Focus on the Desired Output: Instead of telling o1 how to answer, clearly describe what you want the output to be. Let o1 plan and resolve its own steps, leveraging its autonomous reasoning.

• Define Clear Evaluation Criteria: Develop specific criteria for what constitutes a "good" output so that o1 can evaluate its own output and improve. This moves the LLM-as-Judge into the prompt itself. Ask for one specific output per prompt.

• Be Explicit About Output Format: o1 often defaults to a report-style output with numbered headings. Be clear if you need complete files or other specific formats.

• Manage Context and Expect Latency: Since o1 is not a chat model, it will not respond in real time, like email. Make sure you can manage and see the context you are providing to the model. o1 is better suited to high-latency, long-running tasks.

The Deep Roots of DeepSeek: How It All Began - Recode China AI [Link]

Liang's Visions from his first public interview in May 2023:

AI Development:

• Liang aims to build AGI, not just improve existing models like ChatGPT.
• He prioritizes deep research over quick applications, requiring more resources.
• He sees AI as a way to test ideas about human intelligence, like whether language is key to thought.
• He plans to share DeepSeek’s results publicly to keep AI accessible and affordable.

Company Culture & Innovation:

• He hires based on ability, creativity, and passion, preferring fresh graduates for key roles.
• Employees should have freedom to explore and learn from mistakes.
• It can't be forced or taught.
• A shared pace and curiosity drive the team, not strict rules or KPIs.

Competition:

• Startups can still challenge big companies since AI tech is evolving.
• No one has a clear lead in AI yet.
• LLM applications will become easier, creating startup opportunities for decades.
• AI believers stay in for the long run.
• Unconventional approaches can be a game-changer.

Resources & Funding:

• Securing GPUs and a strong engineering team is crucial.
• Traditional VC funding may not fit DeepSeek’s research-heavy approach.
• Innovation is costly, and some waste is inevitable.
• GPUs are a solid investment as they hold value.

Is DeepSeek the new DeepMind? - AI Supremacy [Link]

Implications for the AI Industry:

• DeepSeek's emergence challenges the dominance of Western AI firms like Google DeepMind, Meta, and OpenAI. The success of DeepSeek suggests that open-source models can outperform proprietary ones. It also calls into question the massive spending on AI infrastructure by Big Tech companies.
• Its cost-effectiveness is causing enterprises to rethink their AI strategies. The availability of high-performing, cheaper models could disrupt the business model of companies that rely on expensive, proprietary models.
• Its achievements indicate that China is becoming a leader in AI, particularly in inference-time compute and compute efficiency. This development raises concerns about America's shrinking lead in artificial intelligence.
• Its open-source approach is seen as essential to keeping AI inclusive and accessible. The ability to run powerful models on a laptop could decentralize AI development and reduce reliance on Big Tech.

Arguments about US vs. China in AI:

• The article suggests that the U.S. is losing its lead in AI innovation due to its focus on "Tycoon capitalism" and protectionist policies. The U.S. government's export controls on semiconductors, while intended to slow China's progress, may be inadvertently fueling China's self-reliance and innovation.
• China has advantages in areas such as manufacturing, go-to-market strategies, talent (STEM programs and ML researchers), and patents. China's progress in various overlapping industries creates a "mutually reinforcing feedback loop". The article implies that Chinese work culture of empowering workers with autonomy and collaboration is a strong contrast to the grueling work schedules, rigid hierarchies, and internal competition that are common in Chinese tech firms.
• The article criticizes the massive AI infrastructure projects in the U.S. (dubbed "Project Oracle") as a scheme by the financial elite to control the future of AI. The author argues that these projects prioritize the interests of Big Tech and the financial elite over those of regular citizens and that these AI infrastructure projects are primarily intended to redistribute wealth globally to the elite.

Concerns about AI's Impact:

• The author acknowledges concerns that AI could lead to wage deflation, particularly in white-collar jobs where AI can automate tasks.
• It questions the assumption that AI will create more jobs than it displaces, noting that AI coding tools could negatively impact software engineers.
• It also raises concerns about the potential for misuse of AI, including the use of AI for "authoritarian" control and as a weapon in trade wars. There are also concerns about the potential for backdoors, Trojans, model inversion attacks, sensitive information inference, and automated social engineering via the release of attractive but cheap AI services.

Additional Info:

• DeepSeek is an offshoot of a quantitative hedge fund, High-Flyer, and is fully funded by them.
• It is noted for being more transparent about its methods compared to some Western AI firms.
• Its mission is to "unravel the mystery of Artificial General Intelligence (AGI) with curiosity". They focus on open-source development, research-driven innovation, and making advanced AI accessible to all.

Monopoly Round-Up: China Embarrasses U.S. Big Tech - BIG by Matt Stoller [Link]

• DeepSeek, a Chinese AI firm, developed cost-effective AI models that rival U.S. models and released them on an open-source basis. This is a significant accomplishment, especially since the U.S. has placed export controls that prevent China from accessing the best chips. DeepSeek's approach focused on efficiency, rather than raw computing power, which challenges the assumption that computing power is the primary competitive barrier in AI. This development is considered embarrassing and threatening to big tech and U.S. security.
• The U.S. has heavily invested in AI, with tech giants spending billions on data centers and infrastructure, betting that these investments will provide a competitive advantage. However, DeepSeek’s success suggests that this approach may be flawed. The sources suggest that the U.S. strategy of denying top chips to China may also be ineffective.
• The sources argue that betting on monopolistic national champions is a disastrous national security strategy. It points out that history shows that monopolies are slow to innovate. The U.S. needs to prioritize competition over protecting monopolies. The sources criticize large U.S. tech firms (Meta, Microsoft, Google, Amazon, Apple) for becoming slothful bureaucracies that are not very good at developing and deploying technology.
• Chinese policy is noted to be more aggressive in forcing competition in some sectors. China's electric vehicle industry is cited as an example of this. The Chinese government's crackdown on its big tech firms and financial sector is also mentioned as a move that has seemingly benefited the economy by driving innovation. The success of companies like ByteDance and DeepSeek is mentioned as evidence of this.
• The sources highlight that U.S. anti-monopoly laws take too long to take effect. It uses the example of the Federal Trade Commission's case against Facebook for its acquisition of Instagram and WhatsApp. This case highlights how companies like Facebook acquire and bury innovative competitors rather than compete. It argues that if Facebook had been broken up, there would be tremendous innovation in social networking.
• The sources express uncertainty about the future of AI, noting it might not live up to expectations. It also notes that the competitive advantages in AI are not as straightforward as previously thought.

In a rare interview for AnYong Waves, a Chinese media outlet, DeepSeek CEO Liang Wenfeng emphasized innovation as the cornerstone of his ambitious vision:

. . . we believe the most important thing now is to participate in the global innovation wave. For many years, Chinese companies are used to others doing technological innovation, while we focused on application monetization—but this isn’t inevitable. In this wave, our starting point is not to take advantage of the opportunity to make a quick profit, but rather to reach the technical frontier and drive the development of the entire ecosystem.

― 7 Implications of DeepSeek’s Victory Over American AI Companies - The Algorithmic Bridge [Link]

"Every job is a bundle of tasks.

Every new technology wave (including the ongoing rise of Gen AI) attacks this bundle.

New technology may substitute a specific task (Automation) or it may complement a specific task (Augmentation)"

Extend this analogy far enough, and you get this:

Once technology has substituted all tasks in a job bundle, it can effectively displace the job itself.

Of course, there are limits to this logic. This can only be true for a small number of jobs, which involve task execution only.

But most jobs require a lot more than mere task execution.

They require ‘getting things done’. They require achievement of objectives, accomplishment of outcomes.

In other words, most jobs involve goal-seeking.

This is precisely why previous generations of technologies haven’t fully substituted most jobs. They chip away at tasks in the job bundle without really substituting the job entirely.

Because humans retain their right to play because of their ability to plan and sequence tasks together to achieve goals.

In most previous instances, technology augments humans far more than automating an entire job away.

And that is because humans possess a unique advantage: goal-seeking.

― Slow-burn AI: When augmentation, not automation, is the real threat - Platforms, AI, and the Economics of BigTech [Link]

AI agents are the first instance of technology directly attacking and substituting goals within a role or a team.

In doing so, they directly impact power dynamics within an organization, empowering some roles and weakening others, empowering some teams and weakening others.

― How AI agents rewire the organization - Platforms, AI, and the Economics of BigTech [Link]

This is a brilliant article.

Goal-seeking, for the first time, can be performed by technology.

1. Scope of the role: Effectively, a goal-seeking AI agent can unbundle a goal from the role. They reduce the scope of the role.
2. Scope of the team: They displace the role entirely in a team if the team can now achieve the same goal using an AI agent.
3. Rebundling of roles: Role B is eliminated not because its tasks were fully substituted by technology, nor because its goals were fully substituted by technology, but because the scope of the role no longer justified a separate role.
4. Reworking power structures: Teams have voting rights on the relevance of Roles. The fewer teams speaking to a role’s contributions, the lower the negotiating power for that role within the organization.
5. Roles unbundle, teams rebundle: this cycle of unbundling and rebundling across roles and teams is inherent to the organization of work. AI isn’t fundamentally changing goal-seeking and resource allocation. It is merely inserting itself into the organization and re-organization of work.

YouTube and Podcasts

2025 Predictions with bestie Gavin Baker - All-In Podcasts [Link]

Interesting discussions about new year predictions. Here is a summary of the predictions:

Chamath Palihapitiya:

• Biggest Political Winner: Fiscal conservatives. He believes austerity will reveal waste and fraud in the US government and that this will spill over to state elections.
• Biggest Political Loser: Progressivism. He predicts a repudiation of class-based identity politics in multiple Western countries.
• Biggest Business Winner: Dollar-denominated stablecoins, which he believes will grow substantially and challenge the dominance of Visa and Mastercard.
• Biggest Business Loser: The "MAG 7" companies will see a drawdown in absolute dollars due to high concentration in the indices. He suggests that these companies may not be able to maintain their high valuations, though they are good businesses.
• Biggest Business Deal: The collapse of traditional auto OEMs and a wave of auto mega-mergers, triggered by Tesla's strong position.
• Most Contrarian Belief: A banking crisis in a major mainline bank, triggered by the total indebtedness of Pax America and the impact of higher interest rates.
• Best Performing Asset: Credit Default Swaps (CDS) as an insurance policy against a potential default event.
• Worst Performing Asset: The software industrial complex, or large, bloated enterprise software companies.
• Most Anticipated Trend: Small, arcane regulatory changes related to the supplemental loss ratio that allow the US to kick the debt can down the road.
• Most Anticipated Media: The enormity of files that will be declassified and released by the Trump administration.
• Prediction Market: The MAG 7 representation in the S&P 500 shrinks below 30%.

David Friedberg:

• Biggest Political Winner: Young political candidates, marking a trend of a shift towards younger leaders.
• Biggest Political Loser: Pro-war neoconservatives. He believes they will lose out to figures like JD Vance and Elon Musk.
• Biggest Business Winner: Autonomous hardware and robotics, citing the rise of humanoid robots and their applications.
• Biggest Business Loser: Old defense and aerospace providers, like Boeing and Lockheed Martin. He predicts a shift towards more tech-oriented and rationalized spending in defense. He also thinks Vertical SaaS companies will struggle as AI replaces their services.
• Biggest Business Deal: Massive funding deals for hardware-based manufacturing buildout in the United States, potentially involving government support.
• Most Contrarian Belief: A dramatic rise in socialist movements in the United States, fueled by economic inequality and disruption from AI.
• Best Performing Asset: Chinese tech stocks or ETFs, based on potential deals between the US and China and the strong fundamentals of Chinese tech companies.
• Worst Performing Asset: Vertical SaaS companies again as AI replaces the practices. Also legacy car companies and real estate because of overbuilding and high debt.
• Most Anticipated Trend: The announcement of buildout of nuclear power in the United States.
• Most Anticipated Media: AI Video Games with dynamic story lines
• Prediction Market: Microsoft, AWS, and Google Cloud Revenue Growth.

Gavin Baker:

• Biggest Political Winner: Trump and centrism; also Gen X and Elder Millennials.
• Biggest Political Loser: Putin, due to Europe rearming, which shifts US resources to the Pacific, and Trump's likely tougher stance.
• Biggest Business Winner: Big businesses that use AI thoughtfully, and the robotics industry, as well as companies that make high bandwidth memory.
• Biggest Business Loser: Government service providers with over 35% of their revenue coming from the US government. He also thinks Enterprise application software will be hurt by AI agents
• Biggest Business Deal: A wave of M&A after a period of inactivity and something significant happening with Intel. Also, he thinks independent AI labs will get acquired.
• Most Contrarian Belief: The US will experience at least one year of greater than 5% real GDP growth due to AI and deregulation. He also thinks frontier AI labs will stop releasing their leading-edge models.
• Best Performing Asset: Companies that make high bandwidth memory (HBM).
• Worst Performing Asset: Enterprise application software.
• Most Anticipated Trend: AI will make more progress per quarter in 2025 than it did per year in 2023 and 2024, due to scaling performance through reasoning, pre-training, and test time compute.
• Most Anticipated Media: Season 2 of 1923
• Prediction Market: US Treasury Market Report on Federal Debt in December 2025 above or below $38 trillion
• UFOs: Believes there is a 25% chance the US government is sitting on knowledge of extraterrestrial life.

Jason Calacanis:

• Biggest Business Winner: Tesla and Google for AI and Robotics
• Biggest Business Loser: Open AI
• Biggest Business Deal: Partnerships between Amazon, Uber, Tesla, and Waymo for autonomy, delivery, and e-commerce
• Most Contrarian Belief: Open AI will lose its lead and its nonprofit-to-for-profit transition and become the number four player in AI.
• Best Performing Asset: MAG 7 stocks
• Worst Performing Asset: Legacy car companies and Real Estate.
• Most Anticipated Trend: Exits and DPI will shower down, along with a surge in M&A and IPOs
• Most Anticipated Media: Legacy media outlets owned by billionaires attempting to steer towards the middle
• Prediction Market: Over or under 750,000 deportations by Trump in the first year of office

Building Anthropic | A conversation with our co-founders - Anthropic [Link]

WTF is Artificial Intelligence Really? | Yann LeCun x Nikhil Kamath | People by WTF Ep #4 - Nikhil Kamath [Link]

The Next Frontier: Sam Altman on the Future of A.I. and Society - New York Times Events [Link]

LA's Wildfire Disaster, Zuck Flips on Free Speech, Why Trump Wants Greenland [Link]

Text, camera, action! Frontiers in controllable video generation - William (Bill) Peebles [Link]

Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) - Latent Space [Link]

The State of AI Startups in 2024 [LS Live @ NeurIPS] - Latent Space [Link]

Best of 2024 in Vision [LS Live @ NeurIPS] - Latent Space [Link]

Red-pilled Billionaires, LA Fire Update, Newsom's Price Caps, TikTok Ban, Jobless MBAs - All-In Podcast [Link]

NVIDIA CEO Jensen Huang Keynote at CES 2025 - NVIDIA [Link]

CES 2025 is the world's biggest tech expo. Each January, CES kicks off the tech year by highlighting everything from groundbreaking gadgets to the processors driving our digital world.

NVIDIA's CES announcements showcased its dominance in the AI chip market while highlighting its bold expansion into emerging, high-growth sectors. By emphasizing robotics, autonomous vehicles, and broader accessibility to AI, NVIDIA demonstrated its commitment to staying central to this wave of innovation.

Highlights:

1. GeForce RTX 50 Series GPUs

   NVIDIA unveiled its latest GeForce RTX 50 series GPUs, powered by the advanced Blackwell architecture and set to launch in January. These GPUs deliver significant improvements in gaming and AI performance, with the flagship RTX 5090 priced at \(\$1,999\) and the RTX 5070 at \(\$549\), surpassing the RTX 4090, which debuted at \(\$1,599\) in 2022.

   The 50 series also introduces DLSS 4, a cutting-edge Deep Learning Super Sampling technology that employs a transformer-based architecture to generate three AI-rendered frames for every traditionally rendered one, enhancing graphics quality and gaming experiences. NVIDIA partnered with Micron to supply memory chips for these GPUs.

   Although GeForce RTX GPUs contributed only 9% of NVIDIA’s revenue in the October quarter, the company’s primary growth continues to come from its Data Center segment, driven by AI demand.

2. AI Advancements

   NVIDIA introduced Nemotron, a new family of AI models derived from Meta’s Llama models, including Llama Nemotron Nano, Super, and Ultra, aimed at advancing AI agent capabilities. CEO Jensen Huang projects that the AI agent market could be worth trillions of dollars.

   Additionally, NVIDIA confirmed that its Blackwell AI accelerators are in full production and are being adopted by leading cloud providers and PC manufacturers, further solidifying its position in AI technology.

3. Robotics and Autonomous Vehicles

   NVIDIA debuted Cosmos, the "world's first physical AI model," designed to advance robotics. Trained on 20 million hours of video, Cosmos is open-licensed on GitHub and integrates seamlessly with NVIDIA’s Omniverse platform to provide physics-based simulations for AI model training in robotics and autonomous systems.

   In partnership with Toyota, NVIDIA is collaborating on developing the automaker's latest autonomous vehicles. Huang sees robotics and autonomous technology as a \(\$1\) trillion market opportunity, expecting NVIDIA’s automotive revenue to grow from \(\$4\) billion in FY25 to \(\$5\) billion in FY26, spanning Data Center and OEM segments.

4. Project DIGITS

   NVIDIA announced Project DIGITS, a personal AI supercomputer aimed at democratizing access to powerful AI tools. Starting at \(\$3,000\), the system features the GB10 Grace Blackwell Superchip, 128GB of unified memory, and up to 4TB of NVMe storage. Users can connect two systems for enhanced processing capabilities.

   Designed for AI researchers and data scientists, Project DIGITS provides a cost-effective solution for building complex AI models without relying on large-scale data center resources.

A not comprehensive summary of NVIDIA's efforts on AI, not a summary of this YouTube video:

1. AI Compute Hardware:

   This category includes the physical processing units that perform the core calculations for AI models. These are primarily GPUs, but also include specialized CPUs and other accelerators.

   Focus: High-performance, parallel processing, low latency, memory bandwidth, energy efficiency for AI workloads.

   Examples:

   NVIDIA A100 Tensor Core GPU NVIDIA A40 Tensor Core GPU NVIDIA A10 Tensor Core GPU NVIDIA H100 Tensor Core GPU NVIDIA L40 GPU NVIDIA L4 GPU NVIDIA B100 "Blackwell" Data Center GPU NVIDIA Grace CPU Superchip GeForce RTX 30 Series (Desktop) - Ampere (Relevance for model development) GeForce RTX 50 Series (Desktop) - Blackwell (Relevance for model development) Project DIGITS - Hardware system (personal AI supercomputer).

2. AI Platforms & Systems:

   This category includes integrated hardware and software solutions designed to simplify the development and deployment of AI applications. It encompasses both edge and data center solutions.

   Focus: Ease of use, scalability, optimized performance for specific AI tasks, deployment solutions.

   Examples:

   NVIDIA DGX A100 System NVIDIA Jetson AGX Xavier NX NVIDIA Jetson Orin NVIDIA Jetson Orin Nano NVIDIA Omniverse Platform

3. AI Software & Development Tools:

   This category includes the software libraries, frameworks, and tools that allow developers to build, train, and deploy AI models. It covers both open source and proprietary tools.

   Focus: Developer productivity, model performance, framework support, customization.

   Examples:

   NVIDIA Merlin (Software Library) NVIDIA NeMo Framework NVIDIA TAO Toolkit

4. AI Applications & Solutions:

   This category focuses on specific, industry-focused AI applications built on top of NVIDIA hardware and software.

   Focus: Pre-built solutions, vertical market expertise, end-to-end solutions.

   Examples: Intelligent Video Analytics (IVA), autonomous vehicle solutions, AI-driven healthcare, generative AI.

5. AI Research and Frameworks

   While related to AI Software and development tools, it deserves its own category because of the open source nature of much of the research based tools and APIs, allowing for community contributions and new technology development.

   Focus: Next-generation tools, advanced research, pushing the limits of AI, new technologies and algorythms.

   Examples: Nemotron NVIDIA FLARE (Federated Learning Application Runtime Environment) NVIDIA Research Publications and Open-Source Projects TensorFlow and PyTorch (With NVIDIA's Extensions)

So, my takeaway was entirely different. It was not a commentary on Masa, or Larry, or Sam. I think all of those three companies are, frankly, very good. It was more a comment that you have to be very careful to protect the president's legacy, if I were them, to make sure that the things that get announced are actually further down the technical spectrum and are actually going to be real. Because if they achieve these things, but it costs you a billion dollars and you only hire 50 people, there's going to be a little bit of egg on the face. And so, that was sort of my own takeaway. I think that the things were decoupled. It just seemed more like marketing and sizzle and kind of hastily put together. I think it would be great if OpenAI builds another incredible model, whatever comes after o3, o4, o5. But it's not clear that you have to spend $500 billion to do it. - Chamath Palihapitiya

― Trump's First Week: Inauguration Recap, Executive Actions, TikTok, Stargate + Sacks is Back! - All-In Podcast [Link]

There's a thing called Jevons Paradox, which kind of speaks to this concept. SAA actually tweeted about it. It's an economic concept where, as the cost of a particular use goes down, the aggregate demand for all consumption of that thing goes up. So, the basic idea is that as the price of AI gets cheaper and cheaper, we're going to want to use more and more of it. You might actually get more spending on it in the aggregate. That's right—because more and more applications will become economically feasible. Exactly. That is, I think, a powerful argument for why companies are going to want to continue to innovate on frontier models. You guys are taking a very strong point of view that open source is definitely going to win, that the leading model companies are all going to get commoditized, and therefore, there will be no return on capital—essentially forcing continued innovation on the frontier. - David Sacks

But then there's this dark horse that nobody's talking about—it's called electricity. It's called power. And all these vehicles are electric vehicles. If you said, 'You know, I just did some quick back-of-the-envelope calculations,' if all of the miles in California went to EV ride-sharing, you would need to double the energy capacity of California. Right? Let's not even talk about what it would take to double the energy capacity of the grid and things like that in California. Let's not even go there. Even getting 10% or 20% more capacity is going to be a gargantuan, five-to-ten-year exercise. Look, I live in LA—in a nice area in LA—and we have power outages all the freaking time because the grid is messed up. They're sort of upgrading it as things break. That's literally where we're at in LA, in one of the most affluent neighborhoods. That’s just the reality. So, I think the dark horse, kind of hot take, is combustion engine AVs. Because I don’t know how you can scale AVs really, really massively with the electric grid as it is. - Travis Kalanick

I just wanted to read a message from Brian Yutko, who's the CEO of Wisk, which is building a lot of these autonomous systems. He said: 'First, automatic traffic collision avoidance systems do exist right now. These aircraft will not take control from the pilot to save the aircraft, even if the software and systems on the aircraft know that it’s going to collide. That’s the big flip that needs to happen in aviation—automation can actually kick in and take over, even in piloted aircraft, to prevent a crash. That’s the minimum of where we need to go. Some fighter jets have something called Automatic Ground Collision Avoidance Systems that do exactly this when fighter pilots pass out. It’s possible for commercial aviation as well.' And then, the second thing he said is: 'We need to have better ATC (Air Traffic Control) software and automation. Right now, we use VHF radio communications for safety and critical instructions, and that’s kind of insane. We should be using data links, etc. The whole ATC system runs on 1960s technology. They deserve better software and automation in the control towers—it’s totally ripe for change. The problem is that attempts at reform have failed.' - Chamath Palihapitiya

― DeepSeek Panic, US vs China, OpenAI $40B?, and Doge Delivers with Travis Kalanick and David Sacks - All-In Podcast [Link]

Articles and Blogs

The Art of Leading Teammates - Harvard Business Review [Link]

A Team-Focused Philosophy

• Put the team first, always, even when facing personal adversity.
• Show appreciation for unsung colleagues.
• Set the standard and create a culture of 100% effort.
• Recognize teammates’ individual psychology and the best ways to motivate them.
• Understand and complement the style of the formal leader.
• Recognize and counteract the external forces that can cause selfish behavior.
• Create opportunities to connect as people outside the office.

What Helps—and What Gets in the Way

• The emotions and behaviors that define individuals are formed early.
• Leaders work within a system.
• It can be hard for individual team leaders to influence change across large organizations.
• A leader’s style and influence will take time to evolve.

Early adopters of gen AI can eclipse rivals by using it to identify entirely new product opportunities, automate routine decisions and processes, deliver customized professional services, and communicate with customers more quickly and cheaply than was possible with human-driven processes.

Far from being a source of advantage, even in sectors where its impact will be profound, gen AI will be more likely to erode a competitive advantage than to confer one, because its very nature makes new insights and data patterns almost immediately transparent to anyone using gen AI tools.

If you already have a competitive advantage that rivals cannot replicate using gen AI, the technology may amplify the value you derive from that advantage.

Businesses that try to deny the power of gen AI will certainly fail. Those that adopt it will stay in the fight. But at this stage it looks likely that the only ones that will actually win with it will be those that can apply it to amplify the advantages they already have.

― AI Won’t Give You a New Sustainable Advantage - Harvard Business Review [Link]

To prevent this problem:	Ask about this:	Sample questions:
Conflating Correlation And Causation	Approach to determining causality	Was this analysis based on an experiment? If not, are there confounders (variables that affect the independent and dependent variables)?To what extent were they addressed in the analysis?
Misjudging The Potential Magnitude Of Effects	Sample size and the precision of the results	What was the average effect of the change? What was the sample size and the confidence interval (or range of likely values the true effect would fall into, and the degree to which one is certain it would fall into that range)? How would our course of action change, depending on where the true effect might lie?
A Disconnect Between What Is Measured And What Matters	Outcome measures	What outcomes were measured? Were they broad enough? Did they capture key intended and unintended consequences? Were they tracked for an appropriate period of time? Were all relevant outcomes reported? How do we think they map to broader organizational goals?
Misjudging Generalizability	Empirical setting and subgroup analysis	How similar is the setting of this study to our business context? Does the context or time period of the analysis make it more or less relevant to our decision? What is the composition of the sample being studied, and how does it influence the applicability of the results? Does the effect vary across subgroups or settings? Does this tell us anything about the generalizability of the results?
Overweighting A Specific Result	Broader evidence and further data collection	Are there other analyses that validate the results and approach? What additional data could we collect, and would the benefit of gathering it outweigh the cost of collecting it? How might this change our interpretation of the results?

― Where Data-Driven Decision-Making Can Go Wrong - Harvard Business Review [Link]

Will Psychedelics Propel Your Career? - Harvard Business Review [Link]

Do you want to take a 'trip'? lol

How Scalable Compute Resources Can Boost LLM Performance - HuggingFace [Link]

This blog explains how to scale test-time compute for models like OpenAI's o1 - apply dynamic inference strategies to improve performance without increasing pretraining budgets. These techniques allow smaller models to outperform larger models on tasks such as math problems.

We introduce deliberative alignment, a training paradigm that directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering. We used deliberative alignment to align OpenAI’s o-series models, enabling them to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses.

― Deliberative alignment: reasoning enables safer language models - OpenAI [Link]

Moravec’s paradox is the observation by artificial intelligence and robotics researchers that, contrary to traditional assumptions, reasoning requires very little computation, but sensorimotor and perception skills require enormous computational resources. The principle was articulated by Hans Moravec, Rodney Brooks, Marvin Minsky, and others in the 1980s.

― Common misconceptions about the complexity in robotics vs AI - Harimus Blog [Link]

Yes, as Yann LeCun mentioned in one of his previous campus lectures, LLM might help but it is not the right solution for robotics. This article made several good points:

• Sensorimotor Tasks Are More Complex. The source emphasizes that sensorimotor tasks are harder than many people realize. It was once assumed that perception and action were simple compared to reasoning, but this has turned out to be incorrect. This idea is known as Moravec's Paradox.
• Real-World Interaction is the challenge. Robotics requires robots to interact with a dynamic, chaotic, and complex real world. Tasks that seem simple for humans, like picking up a coffee cup, involve complex, unconscious processes that are hard to program for a robot. Even small changes in the environment can require a complete rewrite of the robot's "move commands". Robots need to break down movements into muscle contractions and forces, which is more complex than it seems.
• Data Requirements is another challenge. LLMs thrive on massive amounts of data, like text and images from the internet. Robotics requires precise, high-quality data that is hard to collect. The variety and preciseness of the data are also important. Unlike LLMs where quantity of data is key, in robotics, the quality of the data collected matters more than the quantity.

Regarding the question "do we need better hardware to learn", I think we need a system of sensors that can capture every physical movement of a body and every angle a body can perceive. In terms of a world model, the system needs to be on a larger scale.

OpenAI has created an AI model for longevity science - MIT Technology Review [Link]

OpenAI's success with GPT-4b micro demonstrates the potential of LLMs to go beyond natural language processing and address highly specialized scientific problems. The model's ability to redesign Yamanaka factors to improve their effectiveness by 50x could be a game-changer in stem cell research, accelerating advancements in regenerative medicine. This development highlights a significant milestone in the use of AI for scientific discovery, particularly in the field of protein engineering and regenerative medicine.

A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples.

― Laws of Tech: Commoditize Your Complement - [Link]

This is exactly Meta's strategy initially competing with close-source AI model businesses - commoditize their complements to increase demand for their own products. And there are more examples mentioned in this article.

• Core Concept:

  Products have substitutes and complements. A substitute is an alternative product that can be bought if the first product is too expensive. A complement is a product usually bought together with another product. Demand for a product increases when the price of its complements decreases. Companies strategically try to commoditize their complements to increase demand for their own products. Commoditizing a complement means driving its price down to a point where many competitors offer indistinguishable goods. This strategy allows a company to become a quasi-monopolist and divert the majority of the consumer surplus to themselves.

• How it works:

  A company seeks a chokepoint or quasi-monopoly in a product composed of multiple layers. It dominates one layer of the stack while fostering competition in another layer. This drives down prices in the commoditized layer, increasing overall demand. The company profits from increased demand for its core product while the competitors in the commoditized layer struggle with low margins. The goal is to make a complement free or very cheap, to increase profits elsewhere. This strategy is an alternative to vertical integration.

• Examples of Commoditization:

  • Microsoft commoditized PC hardware by licensing its OS to many manufacturers, making the PC itself a commodity and increasing demand for MS-DOS.
  • IBM commoditized the add-in market by using off-the-shelf parts and documenting the interfaces, allowing other manufacturers to produce add-on cards for their PCs, which increased the demand for PCs.
  • Netscape open-sourced its web browser to commoditize browsers and increase demand for its server software.
  • Various companies contribute to open-source software to commoditize software and increase demand for hardware and IT consulting services.
  • Sun developed Java to make hardware more of a commodity.
  • The Open Game License (OGL) was created to commoditize the Dungeons and Dragons system and drive sales of core rulebooks.

• Open Source as a Strategic Weapon:

  Open source can be a way for companies to commoditize their complements. It allows companies to share development costs and compete with dominant players. It can also neutralize advantages held by competitors and shift the focus of competition. Open sourcing can prevent a single company from locking up a technology.

• Generalization:

  Many products are composed of layers, each necessary but not sufficient for the final product. The final product is valuable, but the distribution of revenue among the different layers is contentious. Commoditizing complements is a way to control the market without vertical integration. The division of revenue is influenced by power plays and market dynamics.

• Additional Examples:

  The sources list many examples of commoditization in various industries, including hardware vs. software, banks vs. merchants, apps vs. OSes, game portals vs. game devs, telecom vs. users, and many more. The examples illustrate the breadth of this strategy across various tech and non-tech sectors. There are examples of companies commoditizing themselves, such as Stability AI, who commoditized image-generation models and saw little profit themselves.

• Counter-Examples:

  • Sun Microsystems' strategy of making both hardware and software a commodity was not successful.
  • Some companies, like Apple, try to control both the hardware and software aspects of their products, which goes against the commoditization strategy.

• Other Factors:

  Antitrust actions can influence companies and prevent them from crushing competitors. Fear of antitrust actions may have stopped Microsoft from crushing Google.

• Consequences:

  The commoditization of complements can lead to intense competition in certain layers of the tech stack. It can also lead to a concentration of power and revenue in the hands of companies that control key chokepoints.

Reports and Papers

Mixtral of Experts [Link]

Key innovation: Sparse Mixture of Experts (SMoE) with TopK=2.

The state of Generative AI and Machine Learning at the end of 2023 - Intel Tiber AI Studio [Link]

Trends and insights of AI development and deployment in the enterprise - a survey result.

Does Prompt Formatting Have Any Impact on LLM Performance? [Link]

Prompt formats significantly affect LLM performance, with differences as high as 40% observed in code translation tasks for GPT-3.5-turbo. Larger models like GPT-4 demonstrate more resilience to prompt format changes.

JSON format outperformed Markdown in certain tasks, boosting accuracy by 42%. GPT-4 models exhibited higher consistency in responses across formats compared to GPT-3.5 models.

Deliberative Alignment: Reasoning Enables Safer Language Models [Link]

Their training methodology has two stages: 1) supervised fine-tuning on (prompt, CoT, output) datasets where CoTs explicitly reference safety policies, 2) high-compute RL using a reward model informed by safety policies, improving reasoning and adherence.

Genesis is a comprehensive physics simulation platform designed for general purpose Robotics, Embodied AI, & Physical AI applications. It is simultaneously multiple things:

• A universal physics engine re-built from the ground up, capable of simulating a wide range of materials and physical phenomena.
• A lightweight, ultra-fast, pythonic, and user-friendly robotics simulation platform.
• A powerful and fast photo-realistic rendering system.
• A generative data engine that transforms user-prompted natural language description into various modalities of data.

― Genesis: A Generative and Universal Physics Engine for Robotics and Beyond [Link]

Agents - Julia Wiesinger, Patrick Marlow, and Vladimir Vuskovic - Google [Link]

Agent AI: Surveying the Horizons of Multimodal Interaction [Link]

Foundations of Large Language Models [Link]

Atlas of Gray Matter Volume Differences Across Psychiatric Conditions: A Systematic Review With a Novel Meta-Analysis That Considers Co-Occurring Disorders [Link]

"Gray matter volume (GMV) differences across major mental disorders" refers to variations in the amount or density of gray matter in the brain when comparing individuals with mental disorders to those without. Gray matter consists of neuronal cell bodies, dendrites, and synapses and is essential for processing information, controlling movements, and supporting higher cognitive functions like memory, attention, and decision-making.

Structural Abnormalities: Mental disorders are often associated with changes in the brain's structure. GMV differences can highlight specific brain regions that are smaller, larger, or differently shaped in individuals with mental disorders.

Neurobiological Insights: Identifying GMV changes helps researchers understand the neurobiological basis of mental disorders and how these changes may contribute to symptoms like mood dysregulation, cognitive impairment, or altered behavior.

Target for Interventions: Understanding these differences can inform treatments such as targeted therapies, neurostimulation, or cognitive training to address the affected brain regions.

From Efficiency Gains to Rebound Effects: The Problem of Jevons’ Paradox in AI’s Polarized Environmental Debate [Link]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Link]

DeepSeek-R1 is an open-source reasoning model that matches OpenAI-o1 in math, reasoning, and code tasks.

News

NVIDIA Project DIGITS, A Grace Blackwell AI Supercomputer on your desk - NVIDIA [Link]

Constellation inks $1 billion deal to supply US government with nuclear power - Yahoo [Link]

Why 2025 will be the year of AI orchestration [Link]

2025 is anticipated to be the year of AI orchestration for several reasons:

• In 2024, there was broad experimentation in AI, particularly with agentic use cases. In 2025, these pilot programs, experiments, and new use cases are expected to converge, leading to a greater focus on return on investment.
• As organizations deploy more AI agents into their workflows, the need for infrastructure to manage them becomes more critical. This includes managing both internal workflows and those that interact with other services.
• Decision-makers, especially those outside of the technology sector, are seeking tangible results from their AI investments. They are moving beyond experimentation and expect to see a return on their investment in 2025.
• There will be a greater emphasis on productivity, which involves understanding how multiple agents can be made more effective. This will require a focus on accuracy and achieving higher productivity.
• Many new orchestration options are emerging to address the limitations of existing tools such as LangChain. Companies are building orchestration layers to manage AI applications. These frameworks are still early in development, and the field is expected to grow.
• There will be a focus on integrating agents across different systems and platforms, such as AWS's Bedrock and Slack, to allow for the transfer of context between platforms.
• The emergence of powerful reasoning models like OpenAI's o3 and Google's Gemini 2.0 will make orchestrator agents more powerful.

Perplexity AI makes a bid to merge with TikTok U.S. - CNBC [Link]

OpenAI, Alphabet Inc.’s Google, AI media company Moonvalley and several other AI companies are collectively paying hundreds of content creators for access to their unpublished videos, according to people familiar with the negotiations.

― YouTubers Are Selling Their Unused Video Footage to AI Companies - Bloomberg [Link]

Stavridis says Trump’s plan for Greenland ‘not a crazy idea’ - The Hill [Link]

California’s Wildfire Insurance Catastrophe - WSJ [Link]

Rising premiums and limited coverage options could significantly impact Californians, particularly in wildfire-prone areas. The article calls out state leadership for failing to adapt policies to address climate-related risks effectively.

“Our robotics team is focused on unlocking general-purpose robotics and pushing towards AG –level intelligence in dynamic, real-world settings. Working across the entire model stack, we integrate cutting-edge hardware and software to explore a broad range of robotic form factors. We strive to seamlessly blend high-level AI capabilities with the physical constraints of physical.“

― OpenAI has begun building out its robotics team - VentureBeat [Link]

It's surprising because I kind of remember in a public interview Sam said he is not going to go hardware as it's going to be as efficient on that as those companies with hardware foundations like Tesla, NVIDIA, Meta, etc. Now, it's hiring its first hardware robotics roles as announced by Caitlin Kalinowski.

OpenAI’s $500B ‘Stargate Project’ could aid Pentagon’s own AI efforts, official says - Breaking Defense [Link]

This article highlights OpenAI's ambitious Stargate Project and its potential impact on both commercial and government sectors, particularly the U.S. Department of Defense (DoD). Stargate represents a bold step in building the next generation of AI infrastructure, and its success could profoundly influence the future of both private AI development and national security capabilities. The collaboration between industry leaders and government stakeholders will be key to overcoming technical and financial hurdles.

Here are key takeaways:

OpenAI's Stargate Project:

• Objective: Build $500 billion worth of AI infrastructure, including new data centers and power solutions, primarily aimed at training and operating large AI models.
• Initial Funding: $100 billion to be deployed immediately, with ongoing development starting in Texas and other potential sites in the U.S.
• Collaborators: Japan-based SoftBank, Oracle, UAE-based MGX, NVIDIA, Microsoft, and Arm.

DoD Implications:

• AI Challenges in Defense: The DoD faces significant bottlenecks in computing power to meet the demands of modern AI applications, from battlefield decision-making to intelligence analysis and coordinating multi-domain operations (CJADC2).
• Reliance on Private Sector: Stargate could provide essential computing power to address the Pentagon's high-tech needs, especially where DoD lacks in-house capacity.
• Field Applications: Supercomputing resources are essential for training and retraining AI models in dynamic environments, such as battlefield conditions where new inputs may arise.

Challenges:

• Energy Demands: Generative AI models like ChatGPT consume immense electricity. The DoD must consider scalable and portable power sources, such as compact nuclear plants.
• Funding Scrutiny: Despite public commitments, concerns about the financial capability of Stargate’s backers, including SoftBank, have raised questions.
• Technical Constraints: Effective use of AI in military applications depends on robust, secure, and reliable infrastructure to handle high-bandwidth connections and avoid vulnerabilities to jamming.

Political and Economic Context:

• The Stargate Project was announced at a high-profile White House event, underscoring its perceived importance to national interests.
• Skepticism from figures like Elon Musk about the financial feasibility of such an enormous project adds to the intrigue surrounding its rollout.

Trump is planning 100 executive orders starting Day 1 on border, deportations and other priorities - AP News [Link]

A new neural-network architecture developed by researchers at Google might solve one of the great challenges for large language models (LLMs): extending their memory at inference time without exploding the costs of memory and compute. Called Titans, the architecture enables models to find and store during inference small bits of information that are important in long sequences.

Titans combines traditional LLM attention blocks with “neural memory” layers that enable models to handle both short- and long-term memory tasks efficiently. According to the researchers, LLMs that use neural long-term memory can scale to millions of tokens and outperform both classic LLMs and alternatives such as Mamba while having many fewer parameters.

― Google’s new neural-net LLM architecture separates memory components to control exploding costs of capacity and compute [Link]

TikTok restoring service after Trump vows to delay ban - AXIOS [Link]

TikTok's response to the Supreme Court decision [Link]

Amazon bought more renewable power last year than any other company - TechCrunch [Link]

Ozempic, Wegovy and other drugs are among 15 selected for Medicare’s price negotiations [Link]

Waymo Finds a Way Around US Restrictions Targeting Chinese Cars [Link]

More Speech and Fewer Mistakes - Meta News [Link]

NVIDIA Cosmos - NVIDIA [Link]

Announcing The Stargate Project - OpenAI [Link]

OpenAI announces the Stargate Project, a \(\$500\) billion effort to create advanced AI infrastructure. The project begins with an immediate \(\$100\) billion deployment for data centers, starting in Texas. It supports OpenAI’s goal of scaling artificial general intelligence (AGI) and training advanced AI models. Focus on high-value fields like personalized medicine and biotechnology.

NVIDIA GPUs power compute-intensive workloads. Oracle provides high-capacity cloud infrastructure. Microsoft Azure supports scalable distributed AI model training.

Introducing Operator - OpenAI [Link]

It's an AI agent that automates tasks directly in a web browser. You can use Operator to complete repetitive tasks like filling out forms, booking travel, or ordering items online. It uses a new model called Computer-Using Agent (CUA), which integrates GPT-4's vision capabilities with reinforcement learning to interact with graphical user interfaces (GUIs).

Introducing Citations on the Anthropic API - Anthropic [Link]

Happy New Year (/≧▽≦)/

There is a lot to talk about Reinforcement Learning from Human Feedback (RLHF). How about starting with Reinforcement Learning (RL) basics.

Warning: Extremely long article ahead :)

Overview

The process of training a model using reinforcement learning from human feedback (RLHF) involves three key steps, as outlined in the paper titled “Training language models to follow instructions with human feedback” by OpenAI.

Reinforcement Learning

Introduction

Reinforcement Learning (RL) is a machine learning approach where an agent learns to make decisions by interacting with an environment to maximize cumulative rewards.

The agent is the decision-maker or learner in the RL framework. It performs actions in the environment and learns from the feedback it receives. The environment represents everything external to the agent that it interacts with. It provides feedback in response to the agent’s actions. The state is a representation of the current situation of the environment as perceived by the agent. An action is a decision or move taken by the agent at each step based on its policy (a mapping from states to actions). The reward is a scalar feedback signal provided by the environment to indicate how good or bad an action was in achieving the agent’s goal.

An RL problem is typically formalized as a Markov Decision Process (MDP), which includes:

• States (\(S\)): The set of all possible situations in which the agent can find itself.
• Actions (\(A\)): The set of all possible moves or decisions available to the agent.
• Transition Dynamics (\(P(s'|s,a)\)): The probability of transitioning to a new state \(s'\) given the current state \(s\) and action \(a\).
• Rewards (\(R(s,a)\)): The immediate feedback received after taking action \(a\) in state \(s\).
• Policy (\(\pi(a|s)\)): A strategy that defines how the agent selects actions based on states.

The goal of RL is to find an optimal policy \(\pi^*\) that maximizes cumulative rewards (also called return). This involves balancing short-term rewards with long-term planning using trial-and-error interactions with the environment.

The challenges arised from the nature of environment and its dynamics are non-stationary environments, stochastic rewards, and random states:

• In non-stationary environments, the dynamics of the environment (e.g., transition probabilities or reward functions) change over time. This forces RL agents to continuously adapt their policies, which can lead to a drop in performance during the readjustment phase and forgetting previously learned policies.
• Stochastic rewards occur when the reward function is probabilistic rather than deterministic. This introduces noise into the feedback signal, making it harder for the agent to discern which actions truly lead to higher rewards.
• Random states refer to situations where the agent’s observations are noisy or partially observable, making it harder to infer the true state of the environment. Such randomness complicates policy learning because the agent may need to rely on memory or belief states (e.g., Partially Observable Markov Decision Processes, POMDPs) to make decisions. It increases the dimensionality and complexity of the state space.

The challenges related to algorithmic design and computational feasibility are:

• RL algorithms require a significant amount of interaction with the environment to learn effectively, making them data-intensive. Many RL algorithms, particularly model-free methods like policy gradient techniques, require a large number of samples to converge.
• RL agents face the exploration-exploitation dilemma, where they need to balance trying new actions to discover potentially better rewards (Exploration) and using known actions that yield high rewards (Exploitation).
• Many RL problems involve enormous state and action spaces, such as games like Go or real-world robotics tasks. The exponential growth of possible states and actions makes it computationally challenging for RL algorithms to find optimal solutions.
• Poorly designed rewards can lead to unintended behaviors (e.g., an agent exploiting loopholes in the reward structure). Sparse or delayed rewards make it difficult for the agent to associate its actions with outcomes.
• RL agents often struggle to generalize learned behaviors across different tasks or environments. Agents trained in specific simulations (e.g., driving simulators) may fail to perform well in real-world scenarios due to differences in dynamics, noise, or variability.
• RL algorithms are highly sensitive to hyperparameter choices (e.g., learning rate, discount factor). Poor tuning can lead to slow convergence or failure to converge at all, making training unpredictable and requiring significant expertise.
• RL agents often use complex models (e.g., deep neural networks), making their decisions difficult to interpret. This lack of transparency is problematic in safety-critical applications like healthcare or autonomous driving, where understanding the reasoning behind decisions is essential.

Multi-Armed Bandit (MAB)

The multi-armed bandit (MAB) problem is a classic RL problem that exemplifies the exploration-exploitation tradeoff. It provides a simplified framework for decision-making under uncertainty.

Here is a simple scenario to help understand the Multi-Armed Bandit (MAB) problem. Imagine a doctor has three types of prescription drugs to treat a particular disease and \(N\) patients to treat. At the beginning, the doctor has no knowledge about which drug is the most effective. The goal is to identify the best action—the drug that can cure the highest number of patients.

To achieve this goal, we can define action values as:

\[ Q_t(a) = E[R_t \mid A_t = a], \]

where: - \(R_t\) is a random variable representing whether a patient is cured (reward), - \(a\) is an action, which in this case corresponds to selecting a specific type of drug for the patients.

The best action is the one that maximizes the expected reward:

\[ a^* = \arg\max_a Q(a). \]

It’s important to note that an expectation, \(E[x]\), is typically calculated as:

\[ E[x] = \sum x p(x), \]

where \(p(x)\) represents the probability distribution of \(x\). However, in real-world scenarios where \(p(x)\) is unknown and data is limited, the expectation can be approximated using sample averages:

\[ E[x] \approx \frac{\sum x}{N}, \]

where \(N\) is the total number of observations of \(x\). This approximation process is known as Monte Carlo estimation.

The action value \(Q_t(a)\) can be estimated by Sample-Average Method using the following formula:

\[ Q_t(a) = \frac{\text{Total rewards received when action } a \text{ was taken before time } t}{\text{Number of times action } a \text{ was taken before time } t}. \]

Mathematically, this can be expressed as:

\[ Q_t(a) = \frac{\sum_{i=1}^{t-1} R_i \cdot I_{A_i = a}}{\sum_{i=1}^{t-1} I_{A_i = a}}, \]

where: - \(R_i\) is the reward received at time step \(i\), - \(I_{A_i = a}\) is an indicator function that equals 1 if action \(a\) was selected at time step \(i\) and 0 otherwise.

The best action can be select by Greedy Approach: \[ a^* = \arg\max_a Q(a). \] In our case as demonstrated in the diagram below, after 4 times, \(Q_t(1)=0.5, Q_t(2)=0.75, Q_t(3)=0.25\), the best action is determined by \(\arg \max_a Q_t(a)\) which is Action \(A_2\) (\(a=2\)).

However, this approach has some drawbacks such as small sample size and non-stationary environment (e.g. patients are in different consitions). An intuitive alternative is to give Action \(A_1\) and Action \(A_3\) more opportunities. This is called Exploration - Exploitation Tradeoff, which means to balance trying new actions to discover potentially better rewards (Exploration) and using known actions that yield high rewards (Exploitation).

A better approach is called Epsilon-Greedy Strategy which is a simple yet effective method for addressing the exploration-exploitation tradeoff in RL. It involves:

1. Exploration: With a probability of \(\epsilon\), the agent chooses a random action, allowing it to explore the environment and gather new information.
2. Exploitation: With a probability of \(1-\epsilon\), the agent selects the action that has the highest estimated reward (greedy action) based on its current knowledge.

In our case, let \(\epsilon = 20\%\), \(Q_t(1)\) and \(Q_t(3)\) each is given \(10\%\), and \(Q_t(2)\) is given \(80\%\). The next round (5th) of action is decided by random sampling \(A_1,A_2,A_3\) with probability of \(10\%, 80\%,10\%\). If the sampled action is \(A_1\) and the reward is \(1\), then its action value is updated to be is \(Q_t'(1) = (0+1+1+0+1)/5 =0.6\).

The pseudo code of Epsilon Greedy Approach is as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Initialize
for a = 1 to K:
    Q(a) <- 0          # Estimated value of each arm
    N(a) <- 0          # Number of times each arm has been pulled

# Epsilon-Greedy Algorithm
for t = 1 to num_turns:
    with probability ε:
        A <- randomly select an arm (exploration)
    otherwise:
        A <- select the arm with the highest Q(a) (exploitation)

    # Pull the selected arm and observe reward R
    R <- Reward(A)

    # Update the estimates for the selected arm
    N(A) <- N(A) + 1
    Q(A) <- Q(A) + (1 / N(A)) * (R - Q(A))  # Incremental update formula

Note that there is a math trick in the incremental updates Q(A) <- Q(A) + (1 / N(A)) * (R - Q(A)). \[ \begin{equation} \begin{aligned} Q_{n+1} &= {1\over n}\sum^n_{i=1}R_i \space \text{ : average reward in the n+1 iteration}\\ &= {1\over n}(R_n + \sum^{n-1}_{i=1}R_i)\\ &= {1\over n}(R_n + (n-1){1\over n-1}\sum^{n-1}_{i=1}R_i)\\ &= {1\over n} (R_n + (n-1)Q_n)\\ &= {1\over n}(R_n+ n \times Q_n - Q_n)\\ &= Q_n + {1\over n} (R_n - Q_n) \end{aligned} \end{equation} \] The higher the \(\epsilon\), the more opportunities given to actions, and the higher average reward.

(Source: Reinforcement Learning by Sutton and Barto, Chapter 2)

Agent

Long Term Goal

The goal of Agent is long-term reward \[ G_t = R_{t+1}+R_{t+2}+R_{t+3}+... \] So the objective is expected reward \(E[G_t]\) \[ E[G_t] = E[R_{t+1}+R_{t+2}+R_{t+3}+...] \] There are different types of agent tasks:

• Episodic Task: Episodic tasks consist of distinct episodes, where each episode has a clear beginning and end. At the end of an episode, the environment resets to a starting state.

• Continuing Task: Continuing tasks involve ongoing interactions with no natural endpoint. The agent interacts with the environment indefinitely. A key challenge in continuing tasks is that the cumulative reward (\(E[G_t]\)) can become unbounded as time progresses. This makes it difficult to optimize an unbounded objective directly.

  To make the objective bounded, a discount factor (\(\gamma\)) is introduced. The discount factor ensures that more weight is given to immediate rewards while gradually reducing the importance of future rewards. This approach stabilizes the optimization process. \(\gamma \in (0,1)\) is a scalar that determines how much future rewards are discounted compared to immediate rewards. In practice, \(\gamma\) is often set close to 1 (e.g., 0.95 or 0.98), allowing the agent to consider long-term rewards while still prioritizing recent ones.

  The following derivations demonstrate how discounting makes the objective bounded. \[ \begin{equation} \begin{aligned} G_t &= \gamma R_{t+1}+\gamma^2 R_{t+2}+\gamma^3R_{t+3}+...+\gamma^{k-1} R_{t+k} ...\\ &=\sum^{\infty}_{k=0}\gamma^k R_{t+k+1}\\ &\leq \sum^{\infty}_{k=0} \gamma^k \times R_{max} \space \text{ ,where }R_{max} = \max\{R_{t+1},R_{t+2},...R_{t+k}\}\\ &=R_{max} \sum^{\infty}_{k=0} \gamma^k \\ &= R_{max} {1\over 1-\gamma} < \infty \end{aligned} \end{equation} \] The value of \(\gamma\) influences how far-sighted or short-sighted the agent is. If \(\gamma\) is large, the agent is far-sighted, meaning it prioritizes long-term rewards over immediate ones. If \(\gamma\) is small, the agent is short-sighted, focusing heavily on immediate rewards while ignoring distant future outcomes.

  The cumulative reward can also be written as follows, representing how the current cumulative reward is determined by the next step's reward and cumulative reward: \[ \begin{equation} \begin{aligned} G_t &= \gamma R_{t+1}+\gamma^2 R_{t+2}+...\\ &=R_{t+1} + \gamma (R_{t+2} + \gamma R_{t+3} + ...)\\ &=R_{t+1} + \gamma G_{t+1} \end{aligned} \end{equation} \]

Policy

The outcome of RL is policy \(\pi\), which is a projection or mapping from input state \(s\) to output action \(a\).

Deterministic Policy: A deterministic policy maps each state to a single, specific action. In other words, given the same state, the agent will always select the same action. Deterministic policies may fail in environments with high uncertainty or noise, as they do not allow for the exploration of alternative actions. \[ \pi(s)=a \] Stochastic Policy: A stochastic policy maps each state to a probability distribution over possible actions. For a given state, the agent selects an action based on this distribution, meaning different actions can be chosen with varying probabilities. Requires learning and maintaining a probability distribution over actions, which can be computationally expensive. \[ \begin{aligned} \pi(a|s) &= P(a|s) \geq 1, \\ &\text{where }P(a|s) \text{ is the probability of selecting action a in state s.}\\ \sum_{a\in A(s)}\pi(a|s)&=1 \end{aligned} \]

Bellman Equations*

State-Value Functions and Action-Value Functions

State-Value Functions: denoted as \(V_\pi(s)\), represents the expected cumulative future rewards starting from a particular state \(s\) and following a specific policy \(\pi\) thereafter. It measures the "goodness" of being in the state \(s\), considering the long-term rewards achievable from that state under policy \(\pi\). It does not depend on specific actions but rather on the overall behavior dictated by the policy. \[ \begin{aligned} V_\pi(s)= E_\pi[G_t|S_t=s], \\ \space G_t=\sum^\infty_{k=0}\gamma^kR_{t+k+1} \end{aligned} \] Action-Value Functions: denoted as \(Q_\pi(s,a)\), represents the expected cumulative future rewards starting from the state \(s\), taking action \(a\), and then following a specific policy \(\pi\) thereafter. It measures the “goodness” of taking action \(a\) in state \(s\), considering both immediate rewards and future rewards achievable under the policy \(\pi\). It provides more granular information than \(V(s)\), as it evaluates specific actions rather than just states. \[ Q_\pi(s,a) = E_\pi[G_t|S_t=s, A_t=a] \] The relationship between the state-value function and the action-value function can be expressed using the following formula: \[ V_\pi(s) =\sum_{a \in A} \pi(a|s) Q_\pi (s,a) \] This equation shows that the value of a state under policy \(\pi\), \(V_\pi(s)\), is the expected value of the action-value function \(Q_\pi(s,a)\), weighted by the probability of taking each action \(a\) in state \(s\) according to policy \(\pi(a|s)\).

State-Value Bellman Equation and Action-Value Bellman Equation

State-Value Bellman Equation:

(Source: The Bellman Equation: simplify our value estimation)

The State-Value Bellman Equation can be written in a recursive form. \[ \begin{aligned} V_\pi(s) &= E_\pi(G_t|S_t=s)\\ &= E_\pi(R_{t+1}+rG_{t+1}|S_t=s)\\ &=\sum_a \pi(a|s) \sum_{s'}\sum_r p(s',r|s,a) [r + V_\pi(s')] \end{aligned} \]

The tree structure below as an example can help understand the recursive property of the State-Value Bellman Equation. Note that an action \(a\) does not necessarily lead to a specific state \(s\), it can result in multiple possible states, each with a certain probability. These probabilities are determined by the environment, which we typically do not have direct access to.

Action-Value Bellman Equation:

(Source: Two types of value-based methods)

The Action-Value Bellman Equation can be written in a recursive form as well: \[ \begin{aligned} Q_\pi(s,a)&= E_\pi[G_t|S_t=s, A_t=a]\\ &=\sum_{s'}\sum_{r} P(s',r|s,a)[r+\gamma\sum_{a'}\pi(a'|s')Q_\pi(s',a')] \end{aligned} \] The tree structure below as an example can help understand the recursive property of the Action-Value Bellman Equation.

The main limitations of Bellman Equation:

• In real-world problems, the number of states can be extremely large, requiring a separate Bellman equation for each state. This results in a system of simultaneous nonlinear equations due to the presence of the max operator, which can be difficult to solve.
• Solving Bellman equations often requires iterative methods and can demand significant computational resources. This is particularly true when seeking high-precision approximations over many iterations.
• In applications like the Bellman-Ford algorithm for finding shortest paths, the presence of negative cycles can pose a problem. If a cycle has a negative total sum of edges, it can lead to an undefined shortest path since iterating through the cycle can indefinitely reduce the path length.
• The Bellman equation is inherently nonlinear because it involves maximizing over possible actions. This nonlinearity can complicate finding solutions, especially when dealing with large state spaces or complex reward structures.

Policy Iteration

In RL, an optimal policy is a policy that maximizes the expected cumulative reward (or return) for an agent across all states in a Markov Decision Process (MDP). This means that the state-value function \(v_{\pi}(s)\) or the action-value function \(q_{\pi}(s,a)\) under the optimal policy is greater than or equal to that of any other policy for all states and actions. \[ \pi^*(s) \geq \pi(s), \forall s \in \text{states} \] In finite MDPs, at least one optimal policy always exists. However, there may be multiple optimal policies that achieve the same maximum expected return.

Policy Iteration is a dynamic programming algorithm used in RL to compute the optimal policy \(\pi^*\) for a Markov Decision Process (MDP). It alternates between two main steps: policy evaluation and policy improvement, iteratively refining the policy until convergence.

The full policy iteration algorithm pseudocode is in the figure below (Source: Sutton & Barto summary chap 04 - Dynamic Programming):

Here is a detailed explanation of this policy iteration's algorithm pseudocode.

Repeat steps until convergence:

1. Policy evaluation: keep current policy \(\pi\) fixed, find value function \(V(\cdot)\).

   Iterate Bellman update until values converge:

   \[ V(s) \leftarrow \sum_{s',r}p(s',r|s,\pi(s))[r+\gamma V(s')] \] The Bellman operator computes future rewards but discounts them by multiplying with \(\gamma\). This ensures that differences in value functions become smaller with each iteration. In other words, Bellman shrinks distances. To see it mathematically,

   The Bellman operator for the state-value function under a fixed policy \(\pi\) is defined as \[ V^{\pi}(s) = r(s, \pi(s)) + \gamma \sum_s P(s|s,\pi(s))V(s') \] This operator updates the value function by combining the immediate reward and the discounted future rewards.

   We compute the difference after applying the operator: \[ \Big|V^{\pi}_1(s)-V^{\pi}_2(s)\Big| = \Big|r(s,\pi(s))+\gamma\sum_s P (s|s,\pi(s))V_1(s')-\Big[r(s,\pi(s))+\gamma\sum_s P (s|s,\pi(s))V_2(s')\Big]\Big| \] Simplify by canceling out the immediate rewards, we get: \[ \Big|V^{\pi}_1(s)-V^{\pi}_2(s)\Big| = \gamma \Big|\sum_s P (s|s,\pi(s))\Big(V_1(s')-V_2(s')\Big)\Big| \] Since \(\gamma<1\), the difference between \(V_1^{\pi}(s)\) and \(V_1^{\pi}(s)\) is always smaller than the difference between \(V_1(s')\) and \(V_1(s')\). Because Bellman operator shrinks distances, it is a contraction mapping and follows the contraction mapping property.

   In summary, Policy evaluation is a contraction mapping for a fixed policy \(\pi\). Policy evaluation converges because it applies a contraction mapping repeatedly to compute the value function for a fixed policy.

2. Policy improvement: find the best action for \(V(\cdot)\) via one-step lookahead.

   During policy improvement, the current policy \(\pi\) is updated by selecting actions that maximize the expected return for each state \(s\).

   \(\pi(s) \leftarrow \arg \max_a \sum_{s',r}p(s',r|s,a)[r+\gamma V(s')]\)

   The intuition behind: \(V^{\pi}(s)\) measures how good it is to start from the state \(s\) and follow policy \(\pi\). By improving the actions selected by the policy, we ensure that the agent transits into states with higher expected cumulative rewards. This iterative process ensures that each new policy improves upon or equals the previous one in terms of total expected rewards.

Overall, the idea of Policy Iteration can be demonstrated in the diagram below. The evaluation process usually takes a long time while the improvement process is usually fast.

The Generalized Policy Iteration (GPI) method can speed up the policy evaluation process. In GPI, policy evaluation process can be approximate or partial (e.g., only a few iterations instead of full convergence). Policy improvement can also be done incrementally or partially. GPI speeds up the process of finding an optimal policy by relaxing the strict requirements of full convergence in policy evaluation.

Monte Carlo Method

Monte Carlo methods are specifically designed for episodic tasks, where each episode eventually terminates, regardless of the actions taken by the agent. An episode refers to a complete sequence of interactions between an agent and the environment, starting from an initial state and ending in a terminal state.

Monte Carlo for Policy Evaluation

Value function and policy updates occur only after an episode is complete. By waiting for the end of an episode, Monte Carlo methods ensure that all rewards following a state are included in the return calculation. Monte Carlo methods learn purely from sampled experience (episodes), without requiring knowledge of transition probabilities or reward models. Episodes allow Monte Carlo methods to handle stochastic environments by averaging returns across multiple episodes.

Here is an example of calculating the state-value functions and action-action functions by Monte Carlo Method once 2 episodes are completed. Given states \(S=[A,B,C,D,E], A=[1,2,3]\), and two episodes \(E_1,E_2\), (Note: \(A:(1,0.4)\) means state \(A\), action \(1\), and reward \(0.4\)) \[ \begin{aligned} &E_1 = \{A:(1,0.4), B:(2,0.5), A:(2,0.6), C:(2,0.1), B:(3,0.8), E:()\}\\ &E_2 = \{B:(2,0.5), A:(1,0.6), C:(2,0.3), A:(1,0.3), C:(2,0.8), E:()\} \end{aligned} \]

1. State-Value Functions Calculation

   We can calculate \(V(A),V(B),V(C),V(D),V(E)\).

   e.g. there are 4 sequences starting from state \(A\), then the state value function is: \[ \begin{aligned} V(A) &= [(0.4+\gamma 0.5 + \gamma^2 0.6 + \gamma^3 0.1 + \gamma^4 0.8)\\ &+(0.6+\gamma 0.1 + \gamma^2 0.8)\\ &+(0.6+\gamma 0.3 + \gamma^2 0.3 + \gamma^4 0.8)\\ &+(0.3+\gamma 0.8)] / 4 \end{aligned} \]

2. Action-Value Functions Calculation

   We can calculate \(Q(A,1),Q(B,2),\cdots\).

   e.g. there are three sequences starting from \(A:(1,)\), then the action value function is \[ \begin{aligned} Q(A,1) &= [(0.4+\gamma 0.5 + \gamma 0.6 + \gamma^3 0.1+ \gamma^4 0.8)\\ &+(0.6+\gamma 0.3 + \gamma^2 0.3 + \gamma^3 0.8)\\ &+(0.3+\gamma 0.8)]/3 \end{aligned} \]

The pseudocode for the above Monte Carlo for Policy Evaluation is as follows (Source: Reinforcement Learning by Sutton and Barto, Chapter 5):

As part of the algorithm, it loops for each step of episode from the end \(T-1\) to the beginning of the episode. This allows for dynamic programming where some values can be stored and do not need to be re-calculated (see a simple demonstration below).

Monte Carlo for Policy Improvement

\[ \pi_{k+1}(s)=\arg \max_a q_{\pi_k}(s,a) \]

Here is an example of updating policy by action-value function. Given states \(S=[A,B,C,D,E], A=[1,2,3], \gamma=0.5\), and a episode \(E\), (Note: \(A:(1,0.7)\) means state \(A\), action \(1\), and reward \(0.7\)) \[ E = \{A:(1,0.7), B:(1,0.4), A:(3,1.5), C:(2,0.1), B:(3,0.7), A:(1,0.3)\}\ \] Through dynamic programming, cumulative reward is \(G_5(A,1)=0.3\), \(G_4(B,3)=0.7+0.5*3=0.85\), \(G_3(C,2)=0.1+0.5*0.85=0.52\), \(G_2(A,3)=1.5+0.52*0.5=1.76\), \(G_1(B,1)=0.4+1.76*0.5=1.28\), \(G_0(A,1)=0.7+1.28*0.5=1.34\). We can maintain three lists to make the algorithm work:

• Return matrix \(Returns(S,A)\), dimension \((S, A)\): It stores cumulative reward values \(G(S=s,A=a)\). One cell can store multiple values as the number of episodes increases.
• \(Q(S,A)\) matrix: It's initialized as random numbers at the beginning. Updated whenever the return matrix is updated. \(Q(S,A)\) is the average value of the corresponding \(Returns(S,A)\).
• \(\pi(s)\) list: It's updating Epsilon values by giving \(1-\epsilon\) probability to the action with highest reward for each state, according to the updated \(Q(S,A)\) matrix. It facilitates Epsilon Greedy Algorithm.

The final updating result of the above example is in the diagram below.

The pseudocode for the above Monte Carlo for Policy Improvement with action-value function is as follows (Source: Reinforcement Learning by Sutton and Barto, Chapter 5):

Main Limitation

• Policy updates occur only after an episode is completed, which can slow down learning compared to methods like Temporal Difference (TD) learning that update incrementally after each step.
• MC methods do not use bootstrapping (i.e., they do not update value estimates based on other estimates). While this avoids bias, it also means MC methods cannot leverage intermediate value estimates, leading to slower convergence.

Temporal Difference Learning

TD leanring focuses on estimating the value function of a given policy by updating value estimates based on the difference between successive predictions, rather than waiting for an entire episode to conclude.

Given \[ \begin{aligned} V(S_t) &\leftarrow V(S_t) + \alpha \Big[G_t - V(S_t)\Big]\\ G_t &= R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... = R_{t+1} + \gamma G_{t+1}\\ V_{\pi}(s) &= E_\pi[G_t|S_t=s] = E_\pi\Big[R_{t+1} + \gamma G_{t+1}| S_t=s\Big] = R_{t+1} + \gamma V_\pi(S_{t+1}) \end{aligned} \] We can derive the core function of TD: \[ V(S_t) \leftarrow V(S_t) + \alpha [R_{t+1} + \gamma V(R_{t+1}) - V(S_t)] \] The pseudocode of TD learning is as follows. (Source: Sutton & Barto summary chap 06 - Temporal Difference Learning)

From table to function approximation

Main limitations of table based methods:

• As the number of state variables increases, the size of the state space grows exponentially.
• Storing a value table becomes impractical for large or continuous state/action spaces due to memory limitations.
• Table-based methods treat each state (or state-action pair) independently. They cannot generalize knowledge from one part of the state space to another, requiring every state to be visited multiple times for accurate learning.
• In large state spaces, it is unlikely that an agent will visit all relevant states/actions frequently enough to converge to an optimal policy within a reasonable time frame.
• Table-based methods are well-suited for small problems but fail to scale to real-world applications such as autonomous driving, robotics, or complex games like Go or StarCraft.

From tabular to parametric functions:

Fit a parametric function to approximate the value function \(V(s)\) which maps states \(s\) to their corresponding value estimates. \[ f(s,\theta) \approx V(s) \] where \[ f(s,\theta)=w^Ts+b \] To optimize this approximation, we minimize the mean squared error (MSE) loss between the observed value \(v(s)\) and predicted \(\hat{v}(s,{w})\) from Monte Carlo. \(\mu(s)\) is the probability distribution over states. This loss ensures that the predicted values are as close as possible to the observed values. \[ \ell = \min \sum_s \mu(s) \Big[v_\pi(s)-\hat{v}(s,{w})\Big]^2 \] The optimal \({w}\) that minimize the loss can be found by batch Gradient Descent (\(\eta\) is learning rate). \[ w \leftarrow w - \eta \nabla \ell(w) \] where \[ \begin{aligned} \nabla \ell(w) &= \nabla \sum_s \mu(s) \Big[v_\pi(s) - \hat{v}(s,{w})\Big]^2\\ &= \sum_s \mu(s) \nabla \Big[v_\pi(s) - \hat{v}(s,{w})\Big]^2\\ &= -2 \sum_s \mu(s) \Big[v_\pi(s) - \hat{v}(s,{w})\Big]\nabla \hat{v}(s,{w}) \end{aligned} \] From Gradient Descent to Stochastic Gradient Descent:

While batch gradient descent computes gradients over the entire dataset (all states), this can be computationally expensive for large-scale problems. Instead, stochastic gradient descent (SGD) updates the parameters incrementally using one observation at a time. Given observations \((S_1, v_\pi(S_1)), (S_2, v_\pi(S_2)), (S_3, v_\pi(S_3)), ...\), SGD performs updates as follows (\(\alpha\) is learning rate). \[ \begin{aligned} {w}_2 &= {w}_1 + \alpha \Big[v_\pi(S_1) - \hat{v}(S_1,{w_1})\Big] \nabla \hat{v}(S_1,{w}_1)\\ {w}_3 &= {w}_2 + \alpha \Big[v_\pi(S_2) - \hat{v}(S_2,{w_2})\Big] \nabla \hat{v}(S_2,{w}_2)\\ &\cdots \end{aligned} \] The algorithm of Gradient Monte Carlo is as follows. (Source: Reinforcement Learning by Sutton and Barto)

PPO Prior

Average Reward

The average reward is an alternative to the commonly used discounted reward framework. It measures the long-term average reward per time step under a given policy, making it particularly suitable for continuing tasks (non-episodic problems) where there is no natural endpoint or terminal state.

The average reward framework is particularly useful for continuing tasks, where:

• The task does not have a natural termination point (e.g., robot navigation, server optimization, or industrial control systems).
• The agent operates indefinitely, and evaluating its performance based on long-term behavior (rather than episodic returns) is more meaningful.

The average reward for a policy is defined as: \[ r(\pi)=\lim_{h \rightarrow \infty} {1\over h}\sum^h_{t=1} E\Big[R_t | S_0, A_{0:t-1} \sim \pi\Big] \] This simple example shows how average reward is calculated:

Differential Return

The differential return in RL is a concept that arises in the average reward framework, particularly for continuing tasks. It measures the cumulative deviation of rewards from the long-term average reward rate, \(r(\pi)\) , under a given policy \(\pi\).

Differential return aligns with the goal of maximizing long-term performance in continuing tasks by focusing on deviations from steady-state behavior. Unlike discounted returns, differential return does not rely on a discount factor \(\gamma\). This avoids biases introduced by choosing an arbitrary discount factor. It is particularly well-suited for tasks with no natural termination, such as robotics or industrial control systems.

The differential return at time step \(t\) is defined as: \[ G_t = R_{t+1} - r(\pi)+R_{t+2} - r(\pi)+R_{t+3} - r(\pi) \] Then the value functions can be rewritten as \[ \begin{aligned} v_\pi(s) = \sum_a \pi(a|s)\sum_{r,s'} p(s',r|s,a) \Big[r-r(\pi) +v_{\pi}(s')\Big]\\ q_\pi(s,a) = \sum_{r,s'} p(s',r|s,a) \Big[r-r(\pi) +\sum_{a'}\pi(a'|s')q_{\pi}(s',a')\Big] \end{aligned} \] Algorithms like Gradient Monte Carlos can be rewritten by using this differential return.

Policy Gradient and REINFORCE

Objective of policy gradient: \[ \begin{aligned} J(\theta) &= \sum_{s\in S}d^{\pi}(s)V^\pi(s) = \sum_{s\in S} d^\pi(s) \sum_{a\in A} \pi_\theta(a|s) Q^\pi(s,a)\\ d^\pi(s) &= \lim_{t\rightarrow \infty}P(s_t=s|s_o,\pi_\theta) \rightarrow \text{ converege (Markov Property)}\\ &\max J(\theta): \\ &\max \sum_{s\in S} d^\pi(s) V^\pi(s) \implies \theta \leftarrow \theta + \nabla_\theta J(\theta) \end{aligned} \] Policy gradient theorem allows us to compute the gradient of the expected return with respect to the parameters of a parameterized policy, enabling optimization through gradient ascent. \[ \begin{aligned} \nabla_\theta J(\theta) &= \nabla _\theta\Big[\sum_{s\in S} d^\pi(s) \sum_{a\in A} \pi_\theta(a|s) Q^\pi(s,a)\Big]\\ &\propto \sum_{s\in S} d^\pi(s) \sum_{a\in A} Q^\pi(s,a) \nabla _\theta\pi_\theta(a|s)\\ &\implies \theta \leftarrow \theta + \eta\Big[\sum_{s\in S} d^\pi(s) \sum_{a\in A} Q^\pi(s,a) \nabla _\theta\pi_\theta(a|s) \Big] \end{aligned} \] Since Monte Carlo involves a sampling step, which requires an expectation form. The gradient derived above can be re-written as follows, supporting gradient estimation. (Recall: \((\ln x)' = 1/x\)) \[ \begin{aligned} \nabla_\theta J(\theta) &\propto \sum_{s\in S} d^\pi(s) \sum_{a\in A} Q^\pi(s,a) \nabla _\theta\pi_\theta(a|s)\\ &=\sum_{s\in S}d^\pi(s) \sum_{a\in A} \pi_\theta(a|s) Q^\pi(s,a) {\nabla_{\theta} \pi_\theta(a|s)\over \pi_\theta(a|s)}\\ &=E_\pi\Big[Q^\pi(s,a)\nabla_\theta \ln\pi_\theta(a|s)\Big]\\ (&=E_{s \sim \pi, a \sim \pi_\theta(a|s)}\Big[Q^\pi(s,a)\nabla_\theta \ln\pi_\theta(a|s)\Big])\\ \end{aligned} \] Given the above theorems, a Reinforce Algorithm - Monte-Carlo Policy-Gradient algorithm is defined as follows. (Source: Sutton & Barto summary chap 13 - Policy Gradient Methods)

A differentiable policy ensures that small changes in \(\theta\) result in smooth changes in the action probabilities \(\pi(a|s,\theta)\). This is crucial for stable and efficient learning. The softmax function is commonly used to parameterize policies in discrete action spaces. The softmax function is smooth and differentiable, enabling gradient-based optimization. Softmax ensures that all actions have non-zero probabilities, promoting exploration during training.

Here the differentiable policy parameterization \(\pi(a|s,{\theta})\) can be defined by \[ \pi(a|s,\theta) = {\exp(h(s,a,\theta)) \over \sum_{b\in A} \exp(h(s,b,\theta))} \] where \(h(s,a,\theta)=w_a^T+b_a\) is a linear or non-linear function representing the preference for action \(a\). The denominator normalizes the probabilities so that they sum to 1.

The log of the softmax function has a convenient derivative that simplifies gradient computation: \[ \nabla_\theta\ln \pi(a,s,\theta) = \nabla_\theta h(s,a,\theta) - \sum_b \pi(b|s,\theta) \nabla_\theta h(s,b,\theta) \]

Main Limitations of REINFORCE

• REINFORCE requires complete episodes to compute the return for each state, as it relies on Monte Carlo estimates of the expected return. This makes it unsuitable for continuing tasks (non-episodic environments) where there is no clear terminal state.
• The gradient estimates in REINFORCE have high variance because they depend on the total return from sampled episodes, which can vary significantly due to stochasticity in the environment and policy.
• Since REINFORCE updates the policy only after completing an episode, it does not make use of intermediate data. This results in poor sample efficiency, requiring a large number of episodes to learn effectively.
• Unlike Temporal Difference (TD) methods, REINFORCE does not use bootstrapping (i.e., it does not update value estimates based on other estimates). It relies solely on complete returns from episodes.
• The algorithm is highly sensitive to the choice of the learning rate. A poorly chosen learning rate can lead to divergence or extremely slow convergence.

Advantage Function

Advantage function employs the idea of differential return. In the REINFORCE algorithm, with advantage function, policy gradient can be re-written as \[ \begin{aligned} \nabla_\theta J(\theta) &=E_\pi\Big[Q^\pi(s,a)\nabla_\theta \ln\pi_\theta(a|s)\Big]\\ &=E_\pi\Big[A^\pi(s,a)\nabla_\theta \ln\pi_\theta(a|s)\Big]\\ \end{aligned} \] where \[ \begin{aligned} A^{\pi}(s,a) &= Q^\pi(s,a)-V^\pi(s)\\ V^\pi(s) &= \sum_{a\in A}\pi(a|s) Q(s,a) \end{aligned} \]

Off-Policy Policy Gradient

Off-policy policy gradient methods allow learning a target policy while using data generated from a different behavior policy. By reusing past experiences and learning from suboptimal actions, off-policy methods can significantly improve sample efficiency. Off-policy learning allows for better exploration strategies since it can incorporate data from various policies, including exploratory ones.

The policy gradient estimate is defined as \[ \nabla_\theta J(\theta) = E_\beta\Big[{\pi_\theta(a|s)\over \beta(a|s)} Q^\pi(s,a) \nabla _\theta \ln \pi_\theta (a|s)\Big] \] \(\beta(a|s)\) refers to the behavior policy that generates the data used for training. The behavior policy is not necessarily the optimal policy we want to learn (the target policy \(\pi(a|s)\)). Instead, it can be any policy that provides useful exploration of the state-action space. \({\pi_\theta(a|s)\over \beta(a|s)}\) is the important weight for sampling.

Trust Region Policy Optimization (TRPO)

TRPO is an advanced policy gradient method in RL designed to optimize policies while ensuring stable and reliable updates. It addresses some of the limitations of traditional policy gradient methods by incorporating a trust region constraint that limits how much the policy can change in a single update. The difference between REINFORCE and TRPO is that TRPO uses off-policy policy gradient and advantage function, as well as a constraint on the Jullback-Leibler (KL) divergence between the old and new policy.

Recall that REINFORCE's objective of policy gradient is: \[ J(\theta) = \sum_{s\in S} d^\pi(s) \sum_{a\in A} \pi_\theta(a|s) Q^\pi(s,a) \] The derivation of TRPO's objective of policy gradient is: \[ \begin{aligned} J(\theta) &=\sum_{s \in S} d^{\pi}(s) \sum_{a\in A} (\pi_\theta(a|s) \hat{A}_{\theta_{old}}(s,a))\\ &=\sum_{s\in S} d^{\pi_{\theta_{old}}} \sum_{a\in A} (\beta(a|s) {\pi_\theta(a|s)\over \beta(a|s)} \hat{A}_{\theta_{old}}(s,a))\\ &=E_{s\sim d^{\pi_{\theta_{old}}}, a \sim \beta}\Big[{\pi_\theta(a|s)\over \beta(a|s)} \hat{A}_{\theta_{old}}(s,a)\Big]\\ &=E_{s\sim d^{\pi_{\theta_{old}}}, a \sim \pi_{\theta_{old}}}\Big[{\pi_\theta(a|s)\over \pi_{\theta_{old}}(a|s)} \hat{A}_{\theta_{old}}(s,a)\Big]\\ E_{s\sim d^{\pi_{\theta_{old}}}}&\Big[D_{KL}\Big(\pi_{\theta_{old}}(\cdot | s)||\pi_\theta(\cdot|s)\Big)\Big] \leq \delta \end{aligned} \] The TRPO constrained optimization is defined as \[ \begin{aligned} &\max E_{s\sim d^{\pi_{\theta_{old}}}, a \sim \pi_{\theta_{old}}}\Big[{\pi_\theta(a|s)\over \pi_{\theta_{old}}(a|s)} \hat{A}_{\theta_{old}}(s,a)\Big]\\ & s.t. E_{s\sim d^{\pi_{\theta_{old}}}}\Big[D_{KL}\Big(\pi_{\theta_{old}}(\cdot | s)||\pi_\theta(\cdot|s)\Big)\Big] \leq \delta \end{aligned} \] One of the main limitations of TRPO is that the constrained optimization problem can be computationally intensive, especially for large state and action spaces.

PPO

PPO Objective

To address the computational expense of the constrained optimization in TRPO, researchers introduced the CLIP objective in policy gradient methods. The CLIP objective simplifies the optimization process while maintaining stable policy updates. Below are the TRPO objective and its corresponding CLIP version: \[ \begin{aligned} J^{TRPO}(\theta) &= E[r(\theta)\hat{A}_{\theta_{old}}(s,a)]\\ J^{CLIP}(\theta) &= E[\min(r(\theta)\hat{A}_{\theta_{old}}(s,a)), \text{clip}(r(\theta),1-\epsilon, 1+\epsilon) \hat{A}_{\theta_{old}}(s,a)] \end{aligned} \] where \[ \begin{aligned} &r(\theta) = {\pi_{\theta (a|s)} \over \pi_{\theta_{old}(a|s)}}\\ &r(\theta) \in [1-\epsilon, 1+ \epsilon], \\ &i.e. 1-\epsilon \leq r(\theta) \leq 1+\epsilon \end{aligned} \] If \(r(\theta) > 1+\epsilon, r(\theta)=1+\epsilon\). If \(r(\theta) < 1-\epsilon, r(\theta) = 1-\epsilon\).

The CLIP objective ensures that the policy ratio \(r(\theta)\) does not deviate too far from 1 (the old policy), thereby limiting large updates to the policy. The term \(J^{CLIP}(\theta)\) takes the minimum of the “unclipped” objective and the “clipped” version. This prevents overly large policy updates by removing the lower bound when \(r(\theta)\) is outside the clipping range.

The Proximal Policy Optimization (PPO) algorithm (Paper: Proximal Policy Optimization Algorithms) extends the CLIP objective by incorporating additional terms for value function optimization and entropy regularization. The full PPO objective is defined as: \[ J^{PPO}(\theta) = E[J^{CLIP}(\theta) - c_1(V_\theta(s)-V_{target})^2 + c_2 H(s,\pi_\theta(\cdot))] \] where

• \(-(V_\theta(s) - V_{target})^2\) is the negative mean squared error (MSE), which we aim to maximize. It minimizes the difference between the predicted value function \(V_\theta(s)\) and the target value \(V_{target}\). The coefficient \(c_2\) controls the tradeoff between policy optimization and value function fitting.
• \(H(s,\pi_\theta(\cdot))\) represents the entropy of the policy. Maximizing entropy encourages exploration by preventing premature convergence to deterministic policies. The coefficient \(c_2\) determines the weight of this entropy term.

Here is a pseudocode of PPO-Clip Algorithm (Source: OpenAI Spinning Up - Proximal Policy Optimization)

PPO Usage

State, Action, and Reward in the Context of LLMs

In the context of LLMs, the components of reinforcement learning are defined as follows:

1. State: The state corresponds to the input prompt or context provided to the language model. It represents the scenario or query that requires a response.
2. Action: The action is the output generated by the language model, i.e., the response or continuation of text based on the given state (prompt).
3. Reward: The reward is a scalar value that quantifies how well the generated response aligns with human preferences or task objectives. It is typically derived from a reward model trained on human feedback.
4. Policy: A policy refers to the strategy or function that maps a given state (input prompt and context) to an action (the next token or sequence of tokens to generate). The policy governs how the LLM generates responses and is optimized to maximize a reward signal, such as alignment with human preferences or task-specific objectives.

Steps of RLHF Using PPO

The RLHF process using PPO involves three main stages:

1. Training a Reward Model: A reward model is trained to predict human preferences based on labeled data. Human annotators rank multiple responses for each prompt, and this ranking data is used to train the reward model in a supervised manner. The reward model learns to assign higher scores to responses that align better with human preferences.

2. Fine-Tuning the LLM with PPO: After training the reward model, PPO is used to fine-tune the LLM. The steps are as follows:

   1. Initialize Policies: Start with a pre-trained LLM as both the policy model (actor) and optionally as the critic for value estimation.

      • The actor is the language model that generates responses (actions) based on input prompts (states).

        For example: Input: “Explain quantum mechanics.” Output: “Quantum mechanics is a branch of physics that studies particles at atomic and subatomic scales.”

      • The critic is typically implemented as a value function, which predicts how good a particular response (action) is in terms of achieving long-term objectives. This model predicts a scalar value for each token or sequence, representing its expected reward or usefulness.

        For example:

        Input: “Explain quantum mechanics.” → “Quantum mechanics is…” Output: A value score indicating how well this response aligns with human preferences or task objectives.

      • Both the actor and critic can be initialized from the same pre-trained LLM weights to leverage shared knowledge from pretraining. However, their roles diverge during fine-tuning: The actor focuses on generating responses. The critic focuses on evaluating those responses.

   2. Collect Rollouts: Interact with the environment by sampling prompts from a dataset. Generate responses (actions) using the current policy. Compute rewards for these responses using the trained reward model.

   3. Compute Advantage Estimates: Use rewards from the reward model and value estimates from the critic to compute advantages: \[ \hat{A}(s, a) = R_t + \gamma V(s_{t+1}) - V(s_t), \] where $ R_t $ is the reward from the reward model.

   4. Optimize Policy with PPO Objective: Optimize the policy using PPO's clipped surrogate objective: \[ J^{CLIP}(\theta) = \mathbb{E}\left[\min\left(r(\theta)\hat{A}(s, a), \text{clip}(r(\theta), 1-\epsilon, 1+\epsilon)\hat{A}(s, a)\right)\right], \] where $ r() = $ is the probability ratio between new and old policies.

   5. Update Value Function: Simultaneously update the value function by minimizing mean squared error between predicted values and rewards: \[ \mathcal{L}_{\text{value}} = \mathbb{E}\left[(V_\theta(s) - R_t)^2\right]. \]

   6. Repeat: Iterate over multiple epochs until convergence, ensuring stable updates by clipping policy changes.

3. Evaluation: Evaluate the fine-tuned LLM on unseen prompts to ensure it generates outputs aligned with human preferences. Optionally, collect additional human feedback to further refine both the reward model and policy.

The following diagrams summarizes the high-level RLHF process with PPO, from preference data creation, to training a reward model, and using reward model in an RL loop to fine tune LLM.

The following workflow chart illustrates the more detailed training process of RLHF with PPO. (Source: Secrets of RLHF in Large Language Models Part I: PPO)

RLHF Training Tricks

There are practical challenges that arise during RLHF training. These challenges stem from the inherent complexities of RL, especially when applied to aligning LLMs with human preferences. Therefore, tricks are essential for addressing the practical limitations of RLHF, ensuring the training process remains efficient, stable, and aligned with human preferences while minimizing the impact of inherent challenges in RL systems. (Source: Secrets of RLHF in Large Language Models Part I: PPO)

DPO

Bradley-Terry and Plackett-Luce Reward Model

The Bradley-Terry (BT) model is a probabilistic model used to compare pairwise preferences. It assumes that each item (e.g., a response or completion) has an intrinsic quality score, and the probability of one item being preferred over another depends on the relative difference in their scores.

Mathematically, the probability of option \(y_1\) being preferred over option \(y_2\) is given by: \[ P(y_1 \succ y_2|x) = {\exp(r(x,y_1)) \over \exp(r(x,y_1)) + \exp(r(x,y_2))} \] The loss of this reward model is \[ L_R(r_{\phi},D) = -E_{(x,y_w,y_l)\sim D} \Big[\log \sigma\Big(r_{\phi}(x,y_w) - r_{\phi}(x,y_l)\Big)\Big] \] However, the BT model has some limitations:

• It assumes transitivity in preferences (if \(A>B\) and \(B>C\), then \(A >C\)), which may not always hold in real-world data.
• It only handles pairwise comparisons and does not naturally extend to rankings involving more than two items.

The Plackett-Luce (PL) model generalizes the Bradley-Terry model to handle rankings of multiple items, not just pairwise comparisons. It models the probability of a ranking as a sequence of choices. The first-ranked item is chosen based on its relative worth compared to all other items. The second-ranked item is chosen from the remaining items, and so on.

Mathematically, for a ranking \(i_1\succ i_2 \succ ... \succ i_J\), the probability is given by: \[ P(i_1\succ i_2 \succ ... \succ i_J ) = \prod^j_{j=1}{\alpha_{i_j}\over \sum^J_{k=j} \alpha_{i_k}} \] where \(\alpha_{i_j}\) is the worth or quality score of item \(i_j\). The denominator normalizes over all remaining items at each step.

The PL model has several advantages over the BT model:

• Unlike BT, which only works with pairwise comparisons, PL can handle rankings involving multiple items.
• PL can accommodate partial rankings (e.g., ranking only the top n items), making it more versatile in scenarios where full rankings are unavailable.
• When human feedback involves ranking multiple responses rather than just picking one as better, PL captures this richer information better than BT.

DPO Objective

The main reason why RLHF with PPO is hard is that it takes a lot of redundant effort. Policy Model is all we need, all other efforts are not necessary. DPO (Direct Preference Optimization) is a novel alternative to traditional RLHF for fine-tuning LLMs. It simplifies the RLHF process by eliminating the need for complex reward models and RL algorithms. Instead, DPO reframes the problem of aligning LLMs with human preferences as a classification problem using human-labeled preference data.

The idea is DPO and difference between DPO and PPO are shown in the figure below (Source: Direct Preference Optimization: Your Language Model is Secretly a Reward Model)

Recall the Bradley-Terry reward model: \[ \begin{aligned} P(y_1 \succ y_2|x) &= {\exp(r(x,y_1)) \over \exp(r(x,y_1)) + \exp(r(x,y_2))}\\ L_R(r_{\phi},D) &= -E_{(x,y_w,y_l)\sim D} \Big[\log \sigma\Big(r_{\phi}(x,y_w) - r_{\phi}(x,y_l)\Big)\Big] \end{aligned} \] RLHF objective is defined as follows. Keep in mind that no matter whether DPO or PPO is used, the objective is always like this. \[ \max_{\pi_\theta} E_{x \sim D, y \sim \pi_\theta(y|x)}\Big[r_{\phi}(x,y) - \beta D_{KL}\big[\pi_\theta(y|x) || \pi_{ref}(y|x)\big]\Big] \] where \(\beta D_{KL}\big[\pi_\theta(y|x) || \pi_{ref}(y|x)\big]\) is a regularization term. When applying RL to NLP, regularization is often needed. Otherwise RL would explore every possible situation and find out hidden tricks which deviate from a language model.

The optimal policy \(\pi_r(y|x)\) that maximizes the objective is \[ \begin{aligned} \pi_r(y|x) &= {1\over Z(x)}\pi_{ref}(y|x)\exp\Big({1\over \beta}r(x,y)\Big)\\ Z(x) &= \sum_y \pi_{ref}(y|x) \exp\Big({1\over \beta}r(x,y)\Big) \end{aligned} \] where \(\pi_r(y|x)\) is a probability distribution.

Based on this optimal policy, we can derive the reward function for the optimal policy \[ r(x,y)=\beta \log{\pi_r(y|x)\over \pi_{ref}(y|x)} + \beta \log Z(x) \] If we put this reward function in the Bradley-Terry model, we obtain a probability of \(y_1\) being prefered to \(y_2\). \[ \begin{aligned} P^*(y_1 \succ y_2|x) &= {\exp^*(r(x,y_1)) \over \exp^*(r(x,y_1)) + \exp^*(r(x,y_2))}\\ &={\exp(\beta \log{\pi^*_r(y_1|x)\over \pi_{ref}(y_1|x)} + \beta \log Z(x)) \over \exp(\beta \log{\pi^*_r(y_1|x)\over \pi_{ref}(y_1|x)} + \beta \log Z(x)) + \exp(\beta \log{\pi^*_r(y_2|x)\over \pi_{ref}(y_2|x)} + \beta \log Z(x))}\\ &={1\over 1+\exp\Big(\beta \log {\pi^*(y_2|x) \over \pi_{ref}(y_2|x)} - \beta\log {\pi^*(y_1|x)\over \pi_{ref}(y_1|x)}\Big)}\\ &=\sigma\Big(\beta \log {\pi^*(y_1|x)\over \pi_{ref}(y_1|x)} - \beta \log {\pi^*(y_2|x)\over \pi_{ref}(y_2|x)}\Big)\\ \end{aligned} \] With this probability, we have DPO's objective function below. We can optimize this loss function by Maximum Likelihood Extimation: \[ L_{DPO}(\pi_\theta; \pi_{ref}) = -E_{(x,y_w,y_l) \sim D} \Big[\log \sigma \Big(\beta \log {\pi_{\theta}(y_w|x)\over \pi_{ref}(y_w|x)} - \beta \log {\pi_{\theta}(y_l|x)\over \pi_{ref}(y_l|x)}\Big)\Big)\Big] \] Key Ideas of DPO Objective:

• DPO's objective aims to increase the likelihood of generating preferred responses over less preferred ones. By focusing directly on preference data, DPO eliminates the need to first fit a reward model that predicts scalar rewards based on human preferences. This simplifies the training pipeline and reduces computational overhead.
• Value functions exist to help reduce the variance of the reward model. In DPO, the value function is not involved because DPO does not rely on a traditional RL framework, such as Actor-Critic methods. Instead, DPO directly optimizes the policy using human preference data as a classification task, skipping the intermediate steps of training a reward model or estimating value functions.
• DPO was originally designed to work with pairwise preference data, however, recent advancements and adaptations have extended its applicability to ranking preference data as well (e.g RankDPO).

DPO paper has provided detailed steps of deriving the gradient of the DPO objective: (Source: Direct Preference Optimization: Your Language Model is Secretly a Reward Model)

The simplified version of the DPO gradient for better understanding is written as follows. Intuitively, when the difference between \(\hat{r}_{\theta}(x, y_l)\) and \(\hat{r}_{\theta}(x, y_w)\) is large, the gradient takes a larger step during optimization. Conversely, when the difference is small, the objective is optimized with a smaller adjustment.

DPO Usage

Here's how DPO is applied step by step:

1. Initial Setup and Supervised Fine-Tuning (SFT): Begin by fine-tuning a pre-trained LLM using supervised learning on a dataset that is representative of the tasks the model will perform. This step ensures the model has a strong foundation in the relevant domain, preparing it for preference-based optimization.

2. Collect Preference Data: Gather human feedback in the form of pairwise preferences or rankings. Annotators evaluate responses generated by the model and indicate which ones they prefer. Construct a dataset of prompts and corresponding preferred and less-preferred responses.

3. Iterative Rounds of DPO

• Sampling and Annotation: In each round, sample a set of responses from the model for given prompts. Collect new preference annotations based on these samples, allowing for dynamic updates to the preference dataset. (Public preference data works as well. Off-policy and on-policy data both work).

• Preference Optimization: Use DPO to adjust the model's outputs based on collected preference data:

• Model Update: Fine-tune the model using this loss function to increase the likelihood of generating preferred responses.

4. Evaluation and Iteration

• Performance Assessment: After each round, evaluate the model’s performance on new prompts to ensure it aligns with human preferences. Use feedback from these evaluations to inform subsequent rounds of sampling and optimization.

• Iterative Refinement: Continue this loop process over multiple rounds, iteratively refining the model's alignment with human preferences through continuous sampling and preference optimization.

DPO Performance

(Source: Direct Preference Optimization: Your Language Model is Secretly a Reward Model)

DPO Objective Pseudocode

\[ L_{DPO}(\pi_\theta; \pi_{ref}) = -E_{(x,y_w,y_l) \sim D} \Big[\log \sigma \Big(\beta \log {\pi_{\theta}(y_w|x)\over \pi_{ref}(y_w|x)} - \beta \log {\pi_{\theta}(y_l|x)\over \pi_{ref}(y_l|x)}\Big)\Big)\Big] \]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import torch.nn.functional as F

def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta):
    """
    pi_logps: policy logprobs, shape (B,)
    ref_logps: reference model logprobs, shape (B,)
    yw_idxs: preferred completion indices in [0, B-1], shape (T,)
    yl_idxs: dispreferred completion indices in [0, B-1], shape (T,)
    beta: temperature controlling strength of KL penalty

    Each pair of (yw_idxs[i], yl_idxs[i]) represents the
    indices of a single preference pair.
    """

    pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs]
    ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs]

    pi_logratios = pi_yw_logps - pi_yl_logps
    ref_logratios = ref_yw_logps - ref_yl_logps

    losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))
    rewards = beta * (pi_logps - ref_logps).detach()

    return losses, rewards

DPO Variants

The key area of research involves developing variants of DPO and conducting theoretical analyses to understand its limitations and potential improvements. This includes exploring different loss functions or optimization strategies that can be applied within the DPO framework.

• One significant area of research focuses on refining the loss function used in DPO. This includes exploring ways to eliminate the need for a reference model, which can simplify the optimization process.

  Examples:

  • ORPO: Monolithic Preference Optimization without Reference Model

  • SimPO: Simple Preference Optimization with a Reference-Free Reward

• Another key direction involves leveraging existing supervised fine-tuning data as preference data for DPO. This strategy aims to enhance the quality of preference data by utilizing high-quality labeled datasets that may already exist from previous SFT processes.

  Examples:

  • Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Main Difficulties in RLHF

Data Collection

In practice, people noticed that the collection of human feedback in the form of the preference dataset is a slow manual process that needs to be repeated whenever alignment criteria change. And there is increasing difficulty in annotating preference data as models become more advanced, particularly because distinguishing between outputs becomes more nuanced and subjective.

• The paper “CDR: Customizable Density Ratios of Strong-over-weak LLMs for Preference Annotation” explains that as models become more advanced, it becomes harder to identify which output is better due to subtle differences in quality. This makes preference data annotation increasingly difficult and subjective.
• Another paper, “Improving Context-Aware Preference Modeling for Language Models,” discusses how the underspecified nature of natural language and multidimensional criteria make direct preference feedback difficult to interpret. This highlights the challenge of providing consistent annotations when outputs are highly sophisticated and nuanced.
• “Less for More: Enhancing Preference Learning in Generative Language Models” also notes that ambiguity among annotators leads to inconsistently annotated datasets, which becomes a greater issue as model outputs grow more complex.

Reward Hacking

Reward hacking is a common problem in reinforcement learning, where the agent learns to exploit the system by maximizing its reward through actions that deviate from the intended goal. In the context of RLHF, reward hacking occurs when training settles in an unintended region of the loss landscape. In this scenario, the model generates responses that achieve high reward scores, but these responses may fail to be meaningful or useful to the user.

In PPO, reward hacking occurs when the model exploits flaws or ambiguities in the reward model to achieve high rewards without genuinely aligning with human intentions. This is because PPO relies on a learned reward model to guide policy updates, and any inaccuracies or biases in this model can lead to unintended behaviors being rewarded. PPO is particularly vulnerable to reward hacking if the reward model is not robustly designed or if it fails to capture the true objectives of human feedback. The iterative nature of PPO, which involves continuous policy updates based on reward signals, can exacerbate this issue if not carefully managed.

DPO avoids explicit reward modeling by directly optimizing policy based on preference data. However, it can still encounter issues similar to reward hacking if the preference data is biased or if the optimization process leads to overfitting specific patterns in the data that do not generalize well. While DPO does not suffer from reward hacking in the traditional sense (since it lacks a separate reward model), it can still find biased solutions that exploit out-of-distribution responses or deviate from intended behavior due to distribution shifts between training and deployment contexts.

• The article "Reward Hacking in Reinforcement Learning" by Lilian Weng discusses how reward hacking occurs when a RL agent exploits flaws or ambiguities in the reward function to achieve high rewards without genuinely learning the intended task. It highlights that in RLHF for language models, reward hacking is a critical challenge, as models might learn to exploit unit tests or mimic biases to achieve high rewards, which can hinder real-world deployment.
• The research "Scaling Laws for Reward Model Overoptimization" explores how optimizing against reward models trained to predict human preferences can lead to overoptimization, hindering the actual objective.
  1. Impact of Policy Model Size: Holding the RM size constant, experiments showed that larger policy models exhibited similar overoptimization trends as smaller models, despite achieving higher initial gold scores. This implies that their higher performance on gold rewards does not lead to excessive optimization pressure on the RM.
  2. Relationship with RM Data Size: Data size had a notable effect on RM performance and overoptimization. Models trained on fewer than ~2,000 comparison labels showed near-chance performance, with limited improvement in gold scores. Beyond this threshold, all RMs, regardless of size, benefited from increased data, with larger RMs showing greater improvements in gold rewards compared to smaller ones.
  3. Scaling Laws for RM Parameters and Data Size: Overoptimization patterns scaled smoothly with both RM parameter count and data size. Larger RMs demonstrated better alignment with gold rewards and less susceptibility to overoptimization when trained on sufficient data, indicating improved robustness.
  4. Proxy vs. Gold Reward Trends: For small data sizes, proxy reward scores deviated significantly from gold reward scores, highlighting overoptimization risks. As data size increased, the gap between proxy and gold rewards narrowed, reducing overoptimization effects.

Note that the KL divergence term in the RLHF objective is intended to prevent the policy from deviating too much from a reference model, thereby maintaining stability during training. However, it does not fully prevent reward hacking. Reward hacking occurs when an agent exploits flaws or ambiguities in the reward model to achieve high rewards without genuinely aligning with human intentions. The KL divergence penalty does not correct these flaws in the reward model itself, meaning that if the reward model is misaligned, the agent can still find ways to exploit it. KL does not directly address whether the actions align with the true objectives or desired outcomes.

QLoRA is never as simple as a single line of code. Let's start from the scaling law...

Neural Scaling Law

In the context of LLMs, a scaling law refers to an empirical relationship that describes how the performance of a model changes as key resources—such as model size (number of parameters), dataset size, and computational power—are scaled up or down. These laws provide insights into how increasing these factors impacts model accuracy, efficiency, and generalization capabilities.

Compute, dataset size, and model size are not independent of each other. Data size and model size together determine compute. The paper "Algorithmic progress in language models" came up with a rule \(C=6ND\) where \(C\) is compute, \(N\) is model size, and \(D\) is data size.

According to the paper "Scaling Laws for Neural Language Models" and "Training Compute-Optimal Large Language Models", below are the key takeaways of scaling laws for LLMs:

• Performance Improvement: Model performance often improves predictably with increases in size, data, and compute, typically following a power-law relationship.
• Diminishing Returns: Beyond certain thresholds, the benefits of scaling diminish, meaning further increases in resources yield smaller performance gains.
• Trade-offs: Effective scaling requires balancing resources like model parameters and training data. For example, the "Chinchilla scaling law" highlights that increasing data size can sometimes yield better results than merely increasing model size in compute-constrained settings.

These observations are critical for LLM research:

• Guidance for Optimization: Scaling laws help researchers allocate resources efficiently and predict the outcomes of scaling efforts, guiding both model design and training strategies. For example, within fixed computational constraints and limited training duration, scaling laws provide a principled approach to determining the optimal model size that minimizes test loss.

• Predicting model performance: As demonstrated in GPT-4 Technical Report, by fitting the scaling law to the loss of smaller models, the loss of a bigger model can be predicted accurately.

  

The scaling law overlooks a critical practical consideration, which can lead to misconceptions. While it suggests that larger models yield better performance, in reality, the primary compute bottleneck lies in inference rather than training. Training compute is often less constrained because training time can be extended, but deployment costs are significantly higher. From a practical standpoint, a more efficient approach is to train a smaller model for an extended period, as this substantially reduces inference compute requirements.

Quantization

Background

As the scaling law suggested, when training a LLM, reducing the number of parameters is probably not an optimal idea for saving computational resource. Luckily, neural nets are robust in low precision, which means lowering precision won't reduce much model performance.

In GTC March 2024 Keynote with NVIDIA CEO Jensen Huang, Jensen stated that NVIDIA has achieved 1000X increase compute power for the past 8 years, faster than Moore’s law. It can be noticed that in the graph they show TFLOPs on FP8 precision in 2022 and TFLOPs on FP4 precision in 2024. This is a trick because it's easier to achieve higher TFLOPs when the precision is lower. And it shows that there is a trend in hardware industry to achieve higher TFLOPs in low precision.

Data Structure - FP32

The IEEE 754 single-precision floating-point format (FP32) represents a 32-bit number in binary form. It is used for approximating real numbers. The FP32 format has three components:

1. Sign bit (1 bit): Indicates whether the number is positive (0) or negative (1).
2. Exponent (8 bits): Encodes the exponent, biased by 127 to allow both positive and negative exponents.
3. Mantissa or Fraction (23 bits): Represents the significant digits of the number.

The formula for FP32 is as follows: \[ \text{Value} = (-1)^{\text{Sign}} \times (1.\text{Mantissa}) \times 2^{\text{Exponent}-127} \] Following this formula, we can calculate FP32 number. Below is an example:

FP32 provides a wider range and higher precision, making it suitable for tasks requiring numerical accuracy, such as training large-scale deep learning models. FP16, with its lower precision, is optimized for speed and memory efficiency. It is particularly effective for inference tasks or mixed-precision training when paired with FP32 for critical calculations.

However, the overflow problem of FP16 arises due to its limited range of representable values. FP16 has a maximum representable value of 65,504 (\(2^{15} \times (2 - \epsilon)\)), which is much smaller compared to FP32's maximum value of approximately \(3.4 \times 10^{38}\). When computations produce results exceeding this range, an overflow occurs, and the value is replaced by infinity (\(\pm \infty\)). Overflow in FP16 can occur during operations like matrix multiplications or summations in deep learning if the intermediate values exceed the maximum representable range. For example, scaling large tensors or performing high-magnitude computations without normalization can easily result in overflow when using FP16. Overflow leads to loss of numerical accuracy and can destabilize training processes in machine learning. It also affects applications like image processing or scientific simulations where precision and stability are critical.

There are some strategies to mitigate this overflow problem:

• Use mixed-precision training. FP16 is used for most computations but critical operations (e.g., gradient accumulation) are performed in FP32 to prevent overflow.
• Normalize inputs and intermediate values to keep them within the representable range of FP16.
• Use alternative formats like BF16, which have a larger dynamic range while maintaining reduced precision.

Googel Brain BF16 uses the same number of exponent bits as FP32 (8 bits), giving it a much larger dynamic range compared to FP16. This means BF16 can represent very large and very small numbers similar to FP32, avoiding underflows and overflows that are common in FP16. Converting from FP32 to BF16 is straightforward because both formats share the same exponent size. The conversion simply involves truncating the mantissa from 23 bits to 7 bits. BF16 uses only 16 bits per value, reducing memory usage by half compared to FP32. This allows for larger batch sizes and models to fit into limited GPU or TPU memory without sacrificing as much numerical range as FP16 does.

Recently, people have started discussing NVIDIA’s FP8 formats (E4M3 and E5M2) as alternatives to BF16 because of their potential to significantly reduce computational and memory costs while maintaining competitive performance in large-scale machine learning tasks. E4M3 offers higher precision, making it suitable for inference and forward-pass computations where precision is critical. E5M2 provides a wider dynamic range, making it ideal for backward-pass computations during training where large gradients can occur. This flexibility allows FP8 to adapt to different stages of training more effectively than BF16.

NVIDIA’s H100 GPUs are specifically designed to support FP8 with optimized Tensor Cores, achieving up to 9x faster training and 30x faster inference compared to previous-generation GPUs using FP16 or BF16. The Hopper architecture dynamically manages precision transitions (e.g., between FP8 and higher-precision formats like FP32), ensuring stability without manual intervention. "Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs" shows that FP8 can deliver similar convergence behavior and accuracy as BF16 in many LLM tasks, with minimal degradation in performance. For inference, FP8 quantization (e.g., E4M3 for KV cache) has been shown to minimally impact accuracy while significantly improving memory efficiency.

However, FP8 comes with challenges such as occasional instability during training (e.g., loss spikes) and sensitivity in certain tasks like code generation or mathematical reasoning. As a result, training LLMs with FP8 precision remains an active area of research and exploration.

Current LLM Training Method in FP8

A training approach has been developed to leverage FP8's efficiency for specific operations while maintaining numerical stability and precision with BF16 for critical components of the model.

During the training process, FP8 is utilized exclusively for computations within the MLP layers, while BF16 is employed for other components of the Transformer architecture, such as Attention, Activation, and Layer Normalization. Both weights and gradients are maintained in BF16 precision.

• In the forward pass, weights in BF16 are converted to FP8 (E4M3) for matrix multiplications within the MLP layers. Once the computation is completed, the results are immediately converted back to BF16.

• In the backward pass, gradients in BF16 are temporarily converted to FP8 (E5M2) when passing through the MLP layers. After the computations are performed, the results are promptly converted back to BF16.

Even when FP8 is used, RAM savings may not be as significant during training because high precision gradients and weights must be maintained in memory to ensure model stability and convergence. The primary benefit of FP8 lies in its ability to reduce memory usage during inference, where weights can be stored in FP8 format, significantly decreasing the memory footprint compared to higher precision formats like FP16 or BF16. Despite this, FP8 is still utilized during training because it allows for faster computations due to its lower precision. This results in accelerated training processes and improved efficiency, especially on hardware optimized for FP8 operations, such as NVIDIA’s H100 GPUs.

Quantization Process

The process of quantization in LLMs refers to a model compression technique that maps high-precision values (e.g., FP32) to lower-precision representations (e.g., INT8 or FP8).

Here is an example of a simple step-by-step quantization from FP16 to INT4:

1. Range Calculation: Determine the range of FP16 values for the weights or activations. This is typically defined by the minimum and maximum values (\([min, max]\)) in the data.

2. Scale Factor and Zero-Point Computation: Compute a scaling factor (S) that maps the FP16 range to the INT4 range (\([-8, 7]\) for signed INT4 or \([0, 15]\) for unsigned INT4). Optionally, calculate a zero-point (Z) to handle asymmetric quantization, where zero in FP16 does not align with zero in INT4.

   The formula for quantization is: \[ x_q = \text{round}\left(\frac{x}{S} + Z\right) \] where \(x_q\) is the quantized INT4 value, \(x\) is the original FP16 value, \(S\) is the scaling factor, and \(Z\) is the zero-point.

3. Quantization: Map each FP16 value to its corresponding INT4 representation using the computed scale factor and zero-point. This step reduces precision but compresses the data significantly.

There are different types of quantization:

• Asymmetric Quantization vs Summetric Quantization
• Uniform Quantization vs Non-uniform Quantization

Quant in General Matrix Multiply (GEMM)

Quantized matrices are stored in memory in their compressed form. During matrix multiplication, these matrices are dequantized back to higher precision (e.g., FP16 or FP32) to perform computations. This process balances memory efficiency with computational precision.

Quantization can be applied at different levels of granularity, which determines how scaling factors are assigned and used. The "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models" paper introduced several quantization granularity techniques, including per-tensor quantization, per-token quantization, and per-channel quantization:

1. Per-Tensor Quantization: A single scaling factor is applied to the entire tensor (e.g., a weight matrix or activation matrix). It is highly memory-efficient since only one scaling factor needs to be stored. But it is not recommended in practice because outlier values can dominate the scaling factor, leading to significant quantization errors for the rest of the tensor.
2. Per-Channel Quantization: Each channel (e.g., each column of a weight matrix or each feature map in activations) has its own scaling factor. Commonly used for weight matrices in neural networks. It mitigates the impact of outliers by isolating them within individual channels, improving quantization accuracy compared to per-tensor methods. But it can introduce computational overhead during dequantization due to the need for multiple scaling factors.
3. Per-Token Quantization: Each token's activations are assigned a unique scaling factor. Typically used for activations in transformer models. It captures token-specific variations in activations, leading to better precision for tasks with dynamic token distributions. Per-token quantization can be computationally expensive and slower because it requires more scaling factors and additional computations.
4. Group-Wise Quantization (GWQ): this method groups multiple channels or tokens together and applies a shared scaling factor across the group. It reduces computational overhead compared to per-channel or per-token quantization while maintaining finer granularity than per-tensor methods. It's often used for both weights and activations to strike a balance between accuracy and efficiency.

QLoRA

Fine-Tuning Cost

Comparing cost of full fine tuning, LoRA fine tuning, and QLoRA fine tuning:

QLoRA's Contributions

Paper: QLoRA: Efficient Finetuning of Quantized LLMs

4-bit NormalFloat Quantitzation

4-bit NormalFloat Quantitzation adopts the idea of Quantile Quantization which is an information-theoretic method that maps values based on quantiles of the weight distribution. It's a method of data compression where data is quantized (reduced to a smaller set of discrete values) in a way that aims to minimize the information entropy of the resulting data, essentially achieving the most efficient representation possible while still introducing some loss in information, making it a "lossy minimum entropy encoding" technique.

To compute the quantile function for 4-bit NormalFloat (NF4) quantization, the process involves mapping cumulative probabilities to quantization levels optimized for normally distributed data. The quantile function is the inverse of the cumulative distribution function (CDF). For example, as shown in the description, if the probability of \(x < 1.2\) is 0.9, then 1.2 is the corresponding quantile for a cumulative probability of 0.9.

With this quantile function, the probability range from 0 to 1 is divided into 16 equal-sized buckets, as 4 bits can represent \(2^4 = 16\) distinct values. The steps are as follows:

1. Divide the Probability Range: The range of cumulative probabilities \([0, 1]\) is divided into 16 equal intervals or "buckets." These intervals represent equal portions of the probability mass.
2. Apply the Quantile Function: For each bucket's cumulative probability value (e.g., \(p_i = \frac{i}{16}\), where \(i \in [1, 15]\)), the corresponding quantile value is computed using the inverse CDF of a standard normal distribution (\(\Phi^{-1}(p_i)\)).
3. Normalize Quantiles: The resulting quantiles are normalized to fit within a predefined range, typically \([-1, 1]\). This ensures that all quantization levels are symmetrically distributed around zero and fall within a compact range suitable for efficient representation.
4. Assign NF4 Values: The normalized quantiles become the 16 discrete values used by NF4 to represent weights or activations in a compressed format. These values are spaced closer together near zero (where most of the normal distribution's probability mass lies) and farther apart at the extremes, optimizing precision where it matters most.

Double Quantization

Double Quantization (DQ) as introduced in paper "QLoRA: Efficient Finetuning of Quantized LLMs" is a memory optimization technique that quantizes the quantization constants themselves to further reduce the memory footprint of LLMs. It involves two quantization steps:

1. The first quantization involves quantizing the original weights of the model into 4-bit NormalFloat (NF4) format. Weights are divided into small blocks (e.g., 64 elements per block), and each block is scaled by a quantization constant (also known as a scaling factor). This constant ensures that the range of values in each block fits within the representable range of NF4. The quantized weights and their corresponding quantization constants are stored. However, these constants (usually in FP32) can add significant memory overhead.

   To calculate the memory overhead: for a block size of 64, storing a 32 bit quantization constant for each block adds \(32/64=0.5\) bits per parameter on average.

2. The second quantization aims to reduce the memory overhead caused by storing quantization constants. Those quantization constants \(c^{FP32}_2\) are further quantized into 8-bit floating-point values (FP8) with a larger block size (e.g., 256 elements per block). This is a summetric quantization where the mean of the first level factors \(c^{FP32}_2\) is subtracted to center their distribution around zero. This reduces their memory footprint while maintaining sufficient precision for scaling operations. Additionally, another set of quantization constants \(c^{FP32}_1\) is introduced to scale these second-level quantized values.

   To calculate the memory savings: after double quantization, the memory footprint per parameter for scaling factors is reduced from \(32/64=0.5\) bits to \(8/64 + 32/(64\times 256)=0.127\) bits per parameter. This results in saving \(0.5-0.127=0.373\) bits per parameter.

The authors of paper "QLoRA: Efficient Finetuning of Quantized LLMs" compared LLaMA models with different 4-bit data types. They show that the NormalFloat data type significantly improves the bit-for-bit accuracy gains compared to regular 4-bit Floats. While Double Quantization only leads to minor gains, it allows for a more fine-grained control over the memory footprint to fit models of certain size (33B/65B) into certain GPUs (24/48GB). This empirical results show that using FP8 for second-level quantization does not degrade model performance, making it an effective trade-off between precision and memory efficiency.

Paged Optimizers

As described in the QLoRA paper, paged optimizers are a memory management innovation that leverages NVIDIA Unified Memory to handle the memory spikes that occur during gradient checkpointing or when processing large mini-batches with long sequence lengths. NVIDIA Unified Memory allows seamless memory sharing between the GPU and CPU. When the GPU runs out of memory during training, optimizer states (e.g., gradients, momentum, or scaling factors) are paged out (evicted) to CPU RAM. These states are paged back into GPU memory only when needed for computations like gradient updates.

Forward and Backward Implementation

Forward:

Backward:

QLora Usage

QLoRA utilizes bitsandbytes for quantization and is seamlessly integrated with Hugging Face's PEFT and transformers libraries, making it user-friendly. To explore the implementation further, let's dive into the QLoRA code and examine the train() function in qlora.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def train():
    hfparser = transformers.HfArgumentParser((
        ModelArguments, DataArguments, TrainingArguments, GenerationArguments
    ))
    model_args, data_args, training_args, generation_args, extra_args = \
        hfparser.parse_args_into_dataclasses(return_remaining_strings=True)
    training_args.generation_config = transformers.GenerationConfig(**vars(generation_args))
    args = argparse.Namespace(
        **vars(model_args), **vars(data_args), **vars(training_args)
    )
    print(args)
    
    checkpoint_dir, completed_training = get_last_checkpoint(args.output_dir)
    if completed_training:
        print('Detected that training was already completed!')

    model, tokenizer = get_accelerate_model(args, checkpoint_dir)

    ......

The get_accelerate_model() function initializes your model and is a crucial component of implementing QLoRA. Notably, within the AutoModelForCausalLM.from_pretrained() method, it loads the quantization configuration through BitsAndBytesConfig. This setup ensures that the model weights are automatically quantized.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

def get_accelerate_model(args, checkpoint_dir):

    if torch.cuda.is_available():
        n_gpus = torch.cuda.device_count()
    if is_ipex_available() and torch.xpu.is_available():
        n_gpus = torch.xpu.device_count()
        
    max_memory = f'{args.max_memory_MB}MB'
    max_memory = {i: max_memory for i in range(n_gpus)}
    device_map = "auto"

    # if we are in a distributed setting, we need to set the device map and max memory per device
    if os.environ.get('LOCAL_RANK') is not None:
        local_rank = int(os.environ.get('LOCAL_RANK', '0'))
        device_map = {'': local_rank}
        max_memory = {'': max_memory[local_rank]}


    if args.full_finetune: assert args.bits in [16, 32]

    print(f'loading base model {args.model_name_or_path}...')
    compute_dtype = (torch.float16 if args.fp16 else (torch.bfloat16 if args.bf16 else torch.float32))
    model = AutoModelForCausalLM.from_pretrained(
        args.model_name_or_path,
        cache_dir=args.cache_dir,
        load_in_4bit=args.bits == 4,
        load_in_8bit=args.bits == 8,
        device_map=device_map,
        max_memory=max_memory,
        quantization_config=BitsAndBytesConfig(
            load_in_4bit=args.bits == 4,
            load_in_8bit=args.bits == 8,
            llm_int8_threshold=6.0,
            llm_int8_has_fp16_weight=False,
            bnb_4bit_compute_dtype=compute_dtype,
            bnb_4bit_use_double_quant=args.double_quant,
            bnb_4bit_quant_type=args.quant_type,
        ),
        torch_dtype=(torch.float32 if args.fp16 else (torch.bfloat16 if args.bf16 else torch.float32)),
        trust_remote_code=args.trust_remote_code,
        use_auth_token=args.use_auth_token
    )
    ......

Other than some necessary components like tokenizer, train() gives an option of LoRA in addition to full finetune. It requires setup of LoRA config and get_peft_model function from peft package.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

def train():
    ......
  if not args.full_finetune:
          model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=args.gradient_checkpointing)

      if not args.full_finetune:
          if checkpoint_dir is not None:
              print("Loading adapters from checkpoint.")
              model = PeftModel.from_pretrained(model, join(checkpoint_dir, 'adapter_model'), is_trainable=True)
          else:
              print(f'adding LoRA modules...')
              modules = find_all_linear_names(args, model)
              config = LoraConfig(
                  r=args.lora_r,
                  lora_alpha=args.lora_alpha,
                  target_modules=modules,
                  lora_dropout=args.lora_dropout,
                  bias="none",
                  task_type="CAUSAL_LM",
              )
              model = get_peft_model(model, config)
    ......

Not every layers are quantized. QLoRA only quantizes linear projection layers. Some layers like Layer Norm is sensitive to precision, so high precision is required.

I finally got time to have some deep dives. Happy Christmas!

RAM Usage During Training

Training large-scale machine learning models e.g. LLMs, requires significant compute resources. Here’s a breakdown of the possible memory usage (RAM) at various stages of the classic training process, based on the pseudocode below:

1
2
3
4
5
6
7
8
9
10
11
12

model = Model()
optimizer = Adam(model.parameters())

for batch, (X, y) in enumerate(dataloader):
    # Compute prediction and loss
    pred = model(X)
    loss = loss_fn(pred, y)

    # Backpropagation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Key Components of Memory Usage

1. Model Parameters: These are the trainable weights of the model, which need to be stored in memory throughout the training process. The size is proportional to the number of parameters in the model.

2. Model Gradients: Gradients for each parameter are computed during backpropagation and stored temporarily for the optimizer to update the weights.

3. Optimizer States: Optimizers like Adam maintain additional states, including:

   • First-order momentum: Tracks the moving average of gradients.

   • Second-order momentum: Tracks the moving average of squared gradients.

   • Both momentum terms have the same size as the model gradients.

4. Activations: Activation outputs from the forward pass are stored for use during backpropagation, where the Hessian matrix is multiplied with the activations. The memory required for activations can be substantial, especially as batch size increases. While the size of parameters, gradients, and optimizer states remains constant, activation memory scales directly with batch size.

5. Other Overheads: Temporary buffers and memory fragmentation during computation also contribute to RAM usage.

Memory Calculation Examples

1. Gradients and Parameters:

   For 70B model, using 32-bit floating-point precision (FP32): \[ 70\times10^9\times4 \text{ bytes}\times2 =521.5\text{GM} \] This accounts for the weights and their corresponding gradients.

2. Optimizer State:

   Adam optimizer requires two additional states (first and second-order momentum), each the same size as the gradients: \[ 70\times10^9\times4 \text{ byte}\times2 =521.5\text{GM} \]

3. Activations:

   For 70B model with a hidden size of 8192, 80 layers, and FP32 precision, each token’s activation memory: \[ 8192\times80\times4\times12 \text{ bytes/token}=30\text{ MB/token} \]

Simple Strategies for Reducing Memory Usage

1. Activation Checkpointing: Instead of storing all activation outputs, recompute activations during backpropagation as needed. This significantly reduces activation memory at the cost of additional compute time.
2. Mixed Precision Training (FP16): Use 16-bit floating-point precision (FP16) instead of FP32 for model weights, gradients, and activations. This halves the memory requirements without substantial accuracy loss when done correctly.

LoRA

Adapters

The original adapter was introduced in 2019 in the paper "Parameter-Efficient Transfer Learning for NLP". It's a small, additional module added to a pre-trained model to adapt it to a new task without significantly changing the original model parameters.

Adapters generally reduce training latency compared to full fine-tuning because only a small number of parameters (those within the adapter modules) are updated during training. This reduction in trainable parameters leads to lower computational overhead and faster convergence in many cases. Additionally, adapters allow for larger batch sizes due to reduced memory usage, which can further accelerate training

However, adapter layers increase inference latency because they are added sequentially and cannot be parallelized. This issue becomes more pronounced with small batch sizes or when using sharded models, such as GPT-2. Techniques like layer pruning or multi-task settings can mitigate but not completely eliminate this latency.

As shown in the experiment results below, inference latent can be significant (Source: LoRA paper):

LoRA Basics

LoRA (Low-Rank Adaptation) was introduced by a Microsoft team in 2021 in the paper LoRA: Low-Rank Adaptation of Large Language Models. The main idea of LoRA is to enable efficient fine-tuning of large pre-trained models by introducing low-rank trainable matrices into the model’s architecture, while keeping the original model weights frozen. This approach significantly reduces the number of trainable parameters and computational requirements compared to full fine-tuning, without compromising performance.

LoRA approximates weight updates in neural networks using low-rank matrix factorization. Instead of updating the full weight matrix \(W\) , it introduces two smaller trainable matrices \(A\) and \(B\) with size \((r \times d)\) and \((d \times r)\). These matrices have much fewer parameters, as their rank \(r\) is much smaller than the dimensions of \(W\). Instead of training \(\Delta W\), LoRA trains the parameters in \(A\) and \(B\). This can be written in formula: \[ h=W_0x + \Delta Wx = W_0x + BAx \] where \(W_0\) is original prerained weight matrix in size \((d\times d)\) which is frozen during training; \(\Delta W\) is in \((d \times d)\) as well computed by \(BA\). \(x\) is a new input with size \((1 \times d)\).

At the start of the training process, the matrix $ A $ is randomly initialized following a normal distribution \(\mathcal{N}(0, \sigma^2)\), while the matrix $ B $ is initialized as a zero matrix. In the initial round, this setup results in $ BA = 0 $, leading to $ h = W_0x $. This initialization strategy ensures stability by preventing significant deviations of $ W_0 $ from its original state.

LoRA is a groundbreaking method with a lot of benefits:

• Parameter Efficiency: By training only the low-rank matrices, LoRA reduces the number of updated parameters resulting in lower memory usage and faster training.
• Frozen Pre-trained Weights: The original pre-trained weights remain unchanged, preserving the model’s general-purpose knowledge and avoiding catastrophic forgetting.
• No Inference Latency Overhead: Unlike adapters, LoRA does not add additional layers to the model. The low-rank matrices can be merged back into the original weight matrix after fine-tuning, ensuring no additional inference latency.
• Versatility: LoRA can be applied to various architectures (e.g. transformers) and tasks, making it a flexible solution for adapting large models like GPT-3 or RoBERTa to specific use cases.

LoRA Usage

The Microsoft developers of LoRA created a Python package called loralib to facilitate the use of LoRA. With this library, any linear layer implemented as nn.Linear() can be replaced by lora.Linear(). This is possible because LoRA is designed to work with any layer involving matrix multiplication. The lora.Linear() module introduces a pair of low-rank adaptation matrices, which are used to modify the original weight matrix by applying a low-rank decomposition.

1
2
3
4
5
6

# ===== Before =====
# layer = nn.Linear(in_features, out_features)
# ===== After ======
import loralib as lora
# Add a pair of low-rank adaptation matrices with rank r=16
layer = lora.Linear(in_features, out_features, r=16)

Before training the model, all non-lora matrix should be fixed and only LoRA matrices should be set as trainable. Training loops can run as usual.

1
2
3
4
5
6
7

import loralib as lora
model = BigModel()
# This sets requires_grad to False for all parameters without the string "lora_" in their names
lora.mark_only_lora_as_trainable(model)
# Training loop
for batch in dataloader:
   ...

When saving model checkpoints during LoRA fine-tuning, only the LoRA-specific parameters need to be saved, not the entire large pre-trained model. This results in significantly smaller checkpoint files and more efficient storage.

1
2
3
4

# ===== Before =====
# torch.save(model.state_dict(), checkpoint_path)
# ===== After =====
torch.save(lora.lora_state_dict(model), checkpoint_path)

Implementation of LoRA - lora.Linear()

Let's take a deep dive into the lora.Linear() source code:

The lora.Linear class builds upon torch.nn.Linear(). It retains the original weight matrix $ W $ as initialized in nn.Linear.__init__(self, in_features, out_features), and introduces two additional LoRA matrices: self.lora_A and self.lora_B. The matrix self.lora_A has dimensions of $ (r, ) $, while self.lora_B has dimensions of $ (, r) $. These matrices are used to adapt the original weight matrix through low-rank decomposition.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

class Linear(nn.Linear, LoRALayer):
    # LoRA implemented in a dense layer
    def __init__(
        self, 
        in_features: int, 
        out_features: int, 
        r: int = 0, 
        lora_alpha: int = 1, 
        lora_dropout: float = 0.,
        fan_in_fan_out: bool = False, # Set this to True if the layer to replace stores weight like (fan_in, fan_out)
        merge_weights: bool = True,
        **kwargs
    ):
        nn.Linear.__init__(self, in_features, out_features, **kwargs)
        LoRALayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout,
                           merge_weights=merge_weights)

        self.fan_in_fan_out = fan_in_fan_out
        # Actual trainable parameters
        if r > 0:
            self.lora_A = nn.Parameter(self.weight.new_zeros((r, in_features)))
            self.lora_B = nn.Parameter(self.weight.new_zeros((out_features, r)))
            self.scaling = self.lora_alpha / self.r
            # Freezing the pre-trained weight matrix
            self.weight.requires_grad = False
        self.reset_parameters()
        if fan_in_fan_out:
            self.weight.data = self.weight.data.transpose(0, 1)

In the forward() function, it implements \(h=W_0x + \Delta Wx = W_0x+ BAx\).

There is a flag variable called self.merge which is use to flag whether it's doing inference or training. Recall that the original weight matrix remaining unchanged during LoRA training is a key feature of the LoRA - pre-trained weights are freezed and instead small, low-rank matrices are trained to approximate updates.

• During inference, if merge_weights is set to True, the low-rank updates self.lora_B @ self.lora_A are added directly to the frozen pre-trained weights (self.weight). This avoids the need for separate computations of LoRA updates during forward passes, improving efficiency.
• During training, if merge_weights is enabled and weights were previously merged, the updates are subtracted from self.weight to revert it to its original frozen state. This ensures that gradients are not incorrectly computed on the merged weights.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

class Linear(nn.Linear, LoRALayer):
    # LoRA implemented in a dense layer
    def __init__(
        self, 
        in_features: int, 
        out_features: int, 
        r: int = 0, 
        lora_alpha: int = 1, 
        lora_dropout: float = 0.,
        fan_in_fan_out: bool = False, # Set this to True if the layer to replace stores weight like (fan_in, fan_out)
        merge_weights: bool = True,
        **kwargs
    ):

      ......
      
    def train(self, mode: bool = True):
        def T(w):
            return w.transpose(0, 1) if self.fan_in_fan_out else w
        nn.Linear.train(self, mode)
        if mode:
            if self.merge_weights and self.merged:
                # Make sure that the weights are not merged
                if self.r > 0:
                    self.weight.data -= T(self.lora_B @ self.lora_A) * self.scaling
                self.merged = False
        else:
            if self.merge_weights and not self.merged:
                # Merge the weights and mark it
                if self.r > 0:
                    self.weight.data += T(self.lora_B @ self.lora_A) * self.scaling
                self.merged = True    
                
			def forward(self, x: torch.Tensor):
        def T(w):
            return w.transpose(0, 1) if self.fan_in_fan_out else w
        if self.r > 0 and not self.merged:
            result = F.linear(x, T(self.weight), bias=self.bias)            
            result += (self.lora_dropout(x) @ self.lora_A.transpose(0, 1) @ self.lora_B.transpose(0, 1)) * self.scaling
            return result
        else:
            return F.linear(x, T(self.weight), bias=self.bias)
          
      ......

This is a collection of some common useful LangChain (v.0.3.3) practices based on my coding experience so far.

LLM Application Development Landscape

Nowadays, LLM applications can be classified into the following categories.

1. Simple LLM Calls

   Applications where LLMs are used directly to answer questions or perform tasks without additional layers of complexity. The focus is on generating responses to prompts or queries. These are straightforward implementations, often used for tasks like content generation, question answering, or summarization.

   Real-world examples:

   • ChatGPT for Q&A: Users input questions, and the model directly generates answers.
   • Copywriting Tools: Applications like Jasper AI create marketing content, blogs, or product descriptions based on user inputs.

2. Vectorstores (RAG)

   Vectorstores are used in Retrieval-Augmented Generation (RAG) applications, where relevant information is retrieved from a database of embeddings (vectorized representations of text) to enhance the LLM's responses. This allows the LLM to work with domain-specific or proprietary knowledge not contained in its training data.

   Real-world examples:

   • Chatbots for Enterprises: A customer support chatbot retrieves relevant product documentation or FAQs stored in a vectorstore to provide accurate responses.
   • Search-Augmented Systems: Google Bard integrates real-time information retrieval to provide up-to-date and contextually relevant responses.

3. Agents

   Agents are LLM-driven systems that execute tasks autonomously or semi-autonomously based on input instructions. They can make decisions, interact with APIs, and manage workflows. Agents often use reasoning frameworks like ReAct (Reasoning and Acting) to decide what steps to take next.

   Real-world examples:

   • Zapier AI Assistant: Automates workflows by taking instructions, analyzing data, and executing API calls or actions across platforms.
   • LangChain Agents: Used for multi-step tasks such as filling out forms, managing databases, or performing calculations.

4. Agents + Vectorstores

   This combines the reasoning and decision-making capabilities of agents with the data retrieval abilities of vectorstores. These systems can autonomously fetch relevant knowledge from vectorstores and execute tasks, enabling advanced applications like AutoGPT. The integration provides both reasoning depth and domain-specific accuracy.

   Real-world examples:

   • AutoGPT: An open-source agent that can generate business plans by researching topics, retrieving relevant information, and autonomously completing subtasks.
   • GPT Engineer: Helps developers by retrieving relevant programming resources and autonomously generating code, debugging, or improving software projects.

Chaining a Simple Prompt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

from dotenv import load_dotenv
from langchain.prompts.prompt import PromptTemplate
from langchain_openai import ChatOpenAI
from langchain_ollama import ChatOllama
from langchain_core.output_parsers import StrOutputParser

load_dotenv()

# define information to be incorporated to the prompt template
information = """
    Elon Reeve Musk (/ˈiːlɒn/; EE-lon; born June 28, 1971) is a businessman and investor. 
    He is the founder, chairman, CEO, and CTO of SpaceX; angel investor, CEO, product architect and former chairman of Tesla, Inc.; owner, chairman and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. 
    He is the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index, and $254 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX.
    A member of the wealthy South African Musk family, Elon was born in Pretoria and briefly attended the University of Pretoria before immigrating to Canada at age 18, acquiring citizenship through his Canadian-born mother. 
    Two years later, he matriculated at Queen's University at Kingston in Canada. Musk later transferred to the University of Pennsylvania, and received bachelor's degrees in economics and physics. 
    He moved to California in 1995 to attend Stanford University. However, Musk dropped out after two days and, with his brother Kimbal, co-founded online city guide software company Zip2. 
    The startup was acquired by Compaq for $307 million in 1999, and, that same year Musk co-founded X.com, a direct bank. X.com merged with Confinity in 2000 to form PayPal.
    In October 2002, eBay acquired PayPal for $1.5 billion, and that same year, with $100 million of the money he made, Musk founded SpaceX, a spaceflight services company. 
    In 2004, he became an early investor in electric vehicle manufacturer Tesla Motors, Inc. (now Tesla, Inc.). He became its chairman and product architect, assuming the position of CEO in 2008. 
    In 2006, Musk helped create SolarCity, a solar-energy company that was acquired by Tesla in 2016 and became Tesla Energy. In 2013, he proposed a hyperloop high-speed vactrain transportation system. 
    In 2015, he co-founded OpenAI, a nonprofit artificial intelligence research company. 
    The following year, Musk co-founded Neuralink—a neurotechnology company developing brain–computer interfaces—and the Boring Company, a tunnel construction company. 
    In 2022, he acquired Twitter for $44 billion. He subsequently merged the company into newly created X Corp. and rebranded the service as X the following year. 
    In March 2023, he founded xAI, an artificial intelligence company.
"""

# create a prompt template
template = """
Given the information {information} about a person, please create:
1. A short summary
2. Two interesting facts about the person.
"""

# incorporate information into prompt
summary_prompt_template = PromptTemplate(
    input_variables=["information"], template=template
)

# create an llm
llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo")
# llm = ChatOllama(model="llama3")

# create a chain
chain = summary_prompt_template | llm | StrOutputParser()

# prompt the model
response = chain.invoke(input={"information": information})

print(response)

Parsing the Output with a Customized Format

Using PydanticOutputParser and user defined output data structure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

from langchain.output_parsers import PydanticOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field, validator
from langchain_openai import OpenAI

llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo")


# define your desired data structure
class Joke(BaseModel):
    setup: str = Field(description="question to set up a joke")
    punchline: str = Field(description="answer to resolve the joke")

    # add custom validation logic easily with Pydantic
    @validator("setup")
    def question_ends_with_question_mark(cls, field):
        if field[-1] != "?":
            raise ValueError("Badly formed question!")
        return field


# set up a parser + inject instructions into the prompt template.
parser = PydanticOutputParser(pydantic_object=Joke)

# create a prompt with query and instruction
prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()},
)

#a query intended to prompt a language model to populate the data structure
prompt_and_model = prompt | llm
output = prompt_and_model.invoke({"query": "Tell me a joke."})
parser.invoke(output)

Data Ingestion to Pinecone Vectorstore (RAG)

Using TextLoader, CharacterTextSplitter, OpenAIEmbeddings, and Pinecone vector database.

Please refer to LangChain text splitter techniques ;text Split by character; text embedding models for more details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore

load_dotenv()

# load data
loader = TextLoader("doc1.txt")
document = loader.load()

# split data
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(document)

# create embedding
embeddings = OpenAIEmbeddings(openai_api_key=os.environ.get("OPENAI_API_KEY"))

# ingest data to vector db
PineconeVectorStore.from_documents(texts, embeddings, index_name=os.environ['INDEX_NAME'])print("finish")

Data Retrieval from Pinecone Vectorestore (RAG)

langchain-ai/retrieval-qa-chat is a ChatPromptTemplate ensuring answers are based solely on the context.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

embeddings = OpenAIEmbeddings()
llm = ChatOpenAI()

# build user query prompt
query = "what is Pinecone in machine learning?"
chain = PromptTemplate.from_template(template=query) | llm

# store query prompt to vector db
vectorstore = PineconeVectorStore(
    index_name=os.environ["INDEX_NAME"], embedding=embeddings
)

# create a retrieval qa prompt
retrieval_qa_chat_prompt = hub.pull("langchain-ai/retrieval-qa-chat")

# create a prompt chain
combine_docs_chain = create_stuff_documents_chain(llm, retrieval_qa_chat_prompt)

# create retrieval chain
retrival_chain = create_retrieval_chain(
    retriever=vectorstore.as_retriever(), combine_docs_chain=combine_docs_chain
)

# execute the retrieval
result = retrival_chain.invoke(input={"input": query})

print(result)

Customized retrieval prompt:

RunnablePassthrough is used to pass through arguments from one step to the next. It allows us to pass on the user's question to the prompt and model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import os

from dotenv import load_dotenv
from langchain_core.prompts import PromptTemplate
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_pinecone import PineconeVectorStore
from langchain_core.runnables import RunnablePassthrough

load_dotenv()

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

embeddings = OpenAIEmbeddings()
llm = ChatOpenAI()

query = "what is Pinecone in machine learning?"
chain = PromptTemplate.from_template(template=query) | llm

vectorstore = PineconeVectorStore(
    index_name=os.environ["INDEX_NAME"], embedding=embeddings
)


template = """
Use the following pieces of context to answer the question at the end. 
If you don't know the answer, you can say "I don't know". Don't make up an answer.
Use three sentences maximum and keep the answer short and to the point.
Always say "thanks for the question" before answering the question.

{context}

Question: {question}

Answer:
"""

custom_rag_prompt = PromptTemplate.from_template(template=template)
rag_chain = (
    {"context": vectorstore.as_retriever() | format_docs, "question": RunnablePassthrough()}
    | custom_rag_prompt
    | llm
)

res = rag_chain.invoke(query)
print(res)

Chat with a PDF (RAG with FAISS)

Using PyPDFLoader, CharacterTextSplitter, OpenAIEmbeddings, and FAISS local vector database.

Please refer to PDF loader ; Langchain FAISS vectorstore; FAISS for more details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import CharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, OpenAI
from langchain_community.vectorstores import FAISS
from langchain.chains.retrieval import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain import hub

pdf_path = "react.pdf"
loader = PyPDFLoader(file_path=pdf_path)
documents = loader.load()
text_splitter = CharacterTextSplitter(
    chunk_size=1000, chunk_overlap=30, separator="\n"
)
docs = text_splitter.split_documents(documents=documents)

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(docs, embeddings)
vectorstore.save_local("faiss_index_react")

new_vectorstore = FAISS.load_local(
    "faiss_index_react", embeddings, allow_dangerous_deserialization=True
)

retrieval_qa_chat_prompt = hub.pull("langchain-ai/retrieval-qa-chat")
combine_docs_chain = create_stuff_documents_chain(
    OpenAI(), retrieval_qa_chat_prompt
)
retrieval_chain = create_retrieval_chain(
    new_vectorstore.as_retriever(), combine_docs_chain
)

res = retrieval_chain.invoke({"input": "Give me the gist of ReAct in 3 sentences"})
print(res["answer"])

Create a ReAct Agent

Using langchain-ai/react-agent-template to build a ReAct prompt.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

from dotenv import load_dotenv
from langchain import hub
from langchain_openai import ChatOpenAI
from langchain.agents import create_react_agent, AgentExecutor
from langchain_experimental.tools import PythonREPLTool

load_dotenv()

# create an instruction
instructions = """You are an agent designed to write and execute python code to answer questions.
You have access to a python REPL, which you can use to execute python code.
If you get an error, debug your code and try again.
Only use the output of your code to answer the question. 
You might know the answer without running any code, but you should still run the code to get the answer.
If it does not seem like you can write code to answer the question, just return "I don't know" as the answer.
"""

# use an ReAct prompt template
base_prompt = hub.pull("langchain-ai/react-agent-template")
prompt = base_prompt.partial(instructions=instructions)

# make use a tool to execute python code
tools = [PythonREPLTool()]

# define a ReAct agent
agent = create_react_agent(
    prompt=prompt,
    llm=ChatOpenAI(temperature=0, model="gpt-4o-mini"),
    tools=tools,
)

# create the ReAct agent executor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# execute the ReAct agent
agent_executor.invoke(
    input={
        "input": """generate and save in current working directory a QRcode
                    that point to https://jokerdii.github.io/di-blog, you have qrcode package installed already"""
    }
)

agent_executor.invoke(
    input={
        "input": """generate and save in current working directory a synthetic csv dataset 
                    with 1000 rows and 2 columns that is about Amazon product description and price."""
    }
)

Using an LangChain Agent for Tasks

create_csv_agent is an AgentExecutor object able to perform operations in CSVs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_experimental.agents.agent_toolkits import create_csv_agent


load_dotenv()

# make use a CSV agent from langchain
csv_agent = create_csv_agent(
    llm=ChatOpenAI(temperature=0, model="gpt-4o-mini"),
    path="episode_info.csv",
    verbose=True,
)

# execute the agent
csv_agent.invoke(
    input={"input": "how many columns are there in file episode_info.csv"}
)
csv_agent.invoke(
    input={
        "input": "print the seasons by ascending order of the number of episodes they have."
    }
)

Creating an ReAct Agent with Multiple Agents Provided as Tools

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

from typing import Any

from dotenv import load_dotenv
from langchain import hub
from langchain_core.tools import Tool
from langchain_openai import ChatOpenAI
from langchain.agents import (
    create_react_agent,
    AgentExecutor,
)
from langchain_experimental.tools import PythonREPLTool
from langchain_experimental.agents.agent_toolkits import create_csv_agent


load_dotenv()


instructions = """You are an agent designed to write and execute python code to answer questions.
You have access to a python REPL, which you can use to execute python code.
You have qrcode package installed
If you get an error, debug your code and try again.
Only use the output of your code to answer the question. 
You might know the answer without running any code, but you should still run the code to get the answer.
If it does not seem like you can write code to answer the question, just return "I don't know" as the answer.
    """

base_prompt = hub.pull("langchain-ai/react-agent-template")
prompt = base_prompt.partial(instructions=instructions)

# define python agent
tools = [PythonREPLTool()]
python_agent = create_react_agent(
    prompt=prompt,
    llm=ChatOpenAI(temperature=0, model="gpt-4-turbo"),
    tools=tools,
)

python_agent_executor = AgentExecutor(agent=python_agent, tools=tools, verbose=True)

# define CSV agent
csv_agent_executor: AgentExecutor = create_csv_agent(
    llm=ChatOpenAI(temperature=0, model="gpt-4"),
    path="episode_info.csv",
    verbose=True,
)

#### router grand agent

# list agent tools
def python_agent_executor_wrapper(original_prompt: str) -> dict[str, Any]:
    return python_agent_executor.invoke({"input": original_prompt})

tools = [
    Tool(
        name="Python Agent",
        func=python_agent_executor_wrapper,
        description="""useful when you need to transform natural language to python and execute the python code,
                        returning the results of the code execution
                        DOES NOT ACCEPT CODE AS INPUT""",
    ),
    Tool(
        name="CSV Agent",
        func=csv_agent_executor.invoke,
        description="""useful when you need to answer question over episode_info.csv file,
                        takes an input the entire question and returns the answer after running pandas calculations""",
    ),
]

# create grand ReAct agent
prompt = base_prompt.partial(instructions="")
grand_agent = create_react_agent(
    prompt=prompt,
    llm=ChatOpenAI(temperature=0, model="gpt-4-turbo"),
    tools=tools,
)
grand_agent_executor = AgentExecutor(agent=grand_agent, tools=tools, verbose=True)

# execute grand ReAct agent and print output
print(
    grand_agent_executor.invoke(
        {
            "input": "which season has the most episodes?",
        }
    )
)

print(
    grand_agent_executor.invoke(
        {
            "input": "Generate and save in current working directory 15 qrcodes that point to `www.udemy.com/course/langchain`",
        }
    )
)

Function / Tool Calling

LangChain provides a standardized interface for connecting tools to models.

• ChatModel.bind_tools(): a method for attaching tool definitions to model calls.
• AIMessage.tool_calls: an attribute on the AIMessage returned from the model for easily accessing the tool aclls the model decided to make.
• create_tool_calling_agent: an agent constsructor that works with ANY model that implements bind_tools and returns tool_calls.

Directly using PythonREPLTool which is already a tool object. Use with caution because Python REPL can execute arbitrary code on the host machine (e.g., delete files, make network requests).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_openai import ChatOpenAI
from langchain_community.tools.tavily_search import TavilySearchResults

load_dotenv()


@tool
def multiply(x: float, y: float) -> float:
    """Multiply 'x' times 'y'."""
    return x * y


prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "you're a helpful assistant"),
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

tools = [TavilySearchResults(), multiply]
llm = ChatOpenAI(model="gpt-4o-mini")

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)

response = agent_executor.invoke(
    {
        "input": "what is the weather in dubai right now? compare it with San Fransisco, output should in in celsious",
    }
)

print(response)

Tool Calling with Langchain

1
2
3
4
5
6
7
8
9
10
11
12

def multiply(a: int, b: int) -> int:
    """Multiply a and b.

    Args:
        a: first int
        b: second int
    """
    return a * b

llm_with_tools = tool_calling_model.bind_tools([multiply])

result = llm_with_tools.invoke("What is 2 multiplied by 3?")

Token Limitation Handling Strategies

when passing documents into the LLM context window, there are three approaches for handling context window limitations:

1. Stuffing: suff all documents into a single prompt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from langchain.chains.combine_documents.stuff import StuffDocumentsChain
from langchain.chains.llm import LLMChain
from langchain.prompts import PromptTemplate

# define prompt
prompt_template = """Write a concise summary of the following:
"{text}"
CONCISE SUMMARY:"""
prompt = PromptTemplate.from_template(prompt_template)

# define LLM chain
llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo-16k")
llm_chain = LLMChain(llm=llm, prompt=prompt)

# define StuffDocumentsChain
stuff_chain = StuffDocumentsChain(llm_chain=llm_chain, document_variable_name="text")
docs = loader.load()
print(stuff_chain.run(docs))

2. Map-reduce: summarize each document on its own in parallel and put them into a final summary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

from langchain.chains import MapReduceDocumentsChain, ReduceDocumentsChain
from langchain_text_splitters import CharacterTextSplitter

# Map
map_template = """The following is a set of documents
{docs}
Based on this list of docs, please identify the main themes 
Helpful Answer:"""
map_prompt = PromptTemplate.from_template(map_template)
map_chain = LLMChain(llm=llm, prompt=map_prompt)
# Reduce
reduce_template = """The following is set of summaries:
{docs}
Take these and distill it into a final, consolidated summary of the main themes. 
Helpful Answer:"""
reduce_prompt = PromptTemplate.from_template(reduce_template)
reduce_chain = LLMChain(llm=llm, prompt=reduce_prompt)
# Combine documents by mapping a chain over them, then combining results
map_reduce_chain = MapReduceDocumentsChain(
    llm_chain=map_chain,
    reduce_documents_chain=reduce_documents_chain,
    document_variable_name="docs",
    return_intermediate_steps=False,
)
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000, chunk_overlap=0)
split_docs = text_splitter.split_documents(docs)
print(map_reduce_chain.run(split_docs))

3. Refine: The refine documents chain constructs a response by looping over the input documents and iteratively updating its answer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

from langchain.chains.summarize import load_summarize_chain
prompt = """
                  Please provide a summary of the following text.
                  TEXT: {text}
                  SUMMARY:
                  """

question_prompt = PromptTemplate(
    template=question_prompt_template, input_variables=["text"]
)

refine_prompt_template = """
              Write a concise summary of the following text delimited by triple backquotes.
              Return your response in bullet points which covers the key points of the text.
              ```{text}```
              BULLET POINT SUMMARY:
              """

refine_template = PromptTemplate(
    template=refine_prompt_template, input_variables=["text"]

# load refine chain
chain = load_summarize_chain(
    llm=llm,
    chain_type="refine",
    question_prompt=question_prompt,
    refine_prompt=refine_prompt,
    return_intermediate_steps=True,
    input_key="input_documents",
    output_key="output_text",
)
result = chain({"input_documents": split_docs}, return_only_outputs=True)

Coreference Resolution

Adding memory to chatbots.

LangChain provides a way to build applications that have memory using LangGraph's persistence. You can enable persistence in LangGraph applications by providing a checkpointer when compiling the graph. Every iteration, LangGraph takes the information and saves it in a DB (PostgreSQL, MySQL, Redis, and MongoDB saver).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph

workflow = StateGraph(state_schema=MessagesState)

# define the function that calls the model
def call_model(state: MessagesState):
    system_prompt = (
        "You are a helpful assistant. "
        "Answer all questions to the best of your ability."
    )
    messages = [SystemMessage(content=system_prompt)] + state["messages"]
    response = model.invoke(messages)
    return {"messages": response}


# define the node and edge
workflow.add_node("model", call_model)
workflow.add_edge(START, "model")

# add simple in-memory checkpointer
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

Langchain has three main strategies to manage state:

1. Simply stuffing previous messages into a chat model prompt.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages(
    [
        SystemMessage(
            content="You are a helpful assistant. Answer all questions to the best of your ability."
        ),
        MessagesPlaceholder(variable_name="messages"),
    ]
)

chain = prompt | model

ai_msg = chain.invoke(
    {
        "messages": [
            HumanMessage(
                content="Translate from English to French: I love programming."
            ),
            AIMessage(content="J'adore la programmation."),
            HumanMessage(content="What did you just say?"),
        ],
    }
)
print(ai_msg.content)

2. The above, but trimming old messages to reduce the amount of distracting information the model has to deal with.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

from langchain_core.messages import trim_messages
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph

# define trimmer
# count each message as 1 "token" (token_counter=len) and keep only the last two messages
trimmer = trim_messages(strategy="last", max_tokens=2, token_counter=len)

workflow = StateGraph(state_schema=MessagesState)


# define the function that calls the model
def call_model(state: MessagesState):
    trimmed_messages = trimmer.invoke(state["messages"])
    system_prompt = (
        "You are a helpful assistant. "
        "Answer all questions to the best of your ability."
    )
    messages = [SystemMessage(content=system_prompt)] + trimmed_messages
    response = model.invoke(messages)
    return {"messages": response}


# define the node and edge
workflow.add_node("model", call_model)
workflow.add_edge(START, "model")

# ddd simple in-memory checkpointer
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

3. More complex modifications like synthesizing summaries for long running conversations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

from langchain_core.messages import HumanMessage, RemoveMessage
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph

workflow = StateGraph(state_schema=MessagesState)


# Define the function that calls the model
def call_model(state: MessagesState):
    system_prompt = (
        "You are a helpful assistant. "
        "Answer all questions to the best of your ability. "
        "The provided chat history includes a summary of the earlier conversation."
    )
    system_message = SystemMessage(content=system_prompt)
    message_history = state["messages"][:-1]  # exclude the most recent user input
    # Summarize the messages if the chat history reaches a certain size
    if len(message_history) >= 4:
        last_human_message = state["messages"][-1]
        # Invoke the model to generate conversation summary
        summary_prompt = (
            "Distill the above chat messages into a single summary message. "
            "Include as many specific details as you can."
        )
        summary_message = model.invoke(
            message_history + [HumanMessage(content=summary_prompt)]
        )

        # Delete messages that we no longer want to show up
        delete_messages = [RemoveMessage(id=m.id) for m in state["messages"]]
        # Re-add user message
        human_message = HumanMessage(content=last_human_message.content)
        # Call the model with summary & response
        response = model.invoke([system_message, summary_message, human_message])
        message_updates = [summary_message, human_message, response] + delete_messages
    else:
        message_updates = model.invoke([system_message] + state["messages"])

    return {"messages": message_updates}


# Define the node and edge
workflow.add_node("model", call_model)
workflow.add_edge(START, "model")

# Add simple in-memory checkpointer
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

Tracing Application with LangSmith

LangSmith traces LLM calls, tool usage, LLM model latency, token count, and cost.

To integrate LangSmith to our application, we need to generate an API key, add it as "LANGCHAIN_API_KEY" in environment variables, install langsmith dependency, setup our environment. Please refer to set up tracing for detailed steps.

LangChain Hub

LangChain Hub is a comprehensive platform that serves as a repository for pre-built components, tools, and configurations designed to accelerate the development of LLM applications. It simplifies the integration of various building blocks—models, prompts, chains, and agents—enabling developers to create robust and scalable applications without starting from scratch.

LangChain Text Splitter Playground

Text Splitter Playground is a user-friendly interface designed to help developers experiment with and fine-tune text-splitting strategies. In many LLM applications, particularly those involving large documents or retrieval-augmented generation (RAG), it is essential to divide text into manageable chunks while preserving context. This tool allows users to optimize the chunking process for their specific needs.

Substack

the more the AV competition globally heats up, and the more large players invest in the technology, including Tesla, the higher the demand for the Uber platform will become for these AV players.

― Uber Technologies – A brilliant business executing to perfection (Quarterly Update) - Rijnberk InvestInsights [Link]

This article predicts Uber to be a massive beneficiary of the AV / Robotaxi revolution. There indeed is a moat.

Where do LLMs spend their FLOPS? - Artificial Fintelligence [Link]

Theoretical Analysis of LLM FLOPS Allocation

• FLOPS Distribution in Decoder Models:

  • Based on a standard decoder model, FLOPS are allocated as follows for each layer:

    6d² for computing Query (Q), Key (K), and Value (V) matrices.

    2d² for computing the attention output matrix, using the formula: softmax(Q @ K.T) @ V.

    16d² for running the feedforward network (FFN).

    This results in a total of 24d² FLOPS per layer.

  • Percentage-wise:

    25% of the time is spent computing QKV.

    ~8% is spent computing the attention output matrix.

    ~66% is spent running the FFN.

• Attention Mechanism:

  While the attention equation itself \(softmax(QK^T/\sqrt{d_{head}})V\) has negligible computational cost (~0.005% for Llama7B) compared to other operations, its impact on memory usage necessitates techniques like KV cache and flash attention.

• KV Cache:

  The KV cache, essential for efficient attention computation, requires O(T) memory, where T is the number of tokens generated.

  The memory size of the KV cache is calculated as 4 * number of layers * d_model bytes.

  While the KV cache demands a significant amount of memory, it essentially reuses the same memory space throughout the token generation process.

• Modern Architectures:

  Architectures like Mistral 7B and Llama2 employ Grouped Query Attention (GQA) and sliding window attention to optimize performance.

  GQA reduces the KV cache size by sharing the KV projection across multiple heads. This leads to a linear decrease in memory consumption as the number of KV heads decreases.

  Sliding window attention limits the KV cache size to the window size (e.g., 4096 for Llama7B), further controlling memory usage.