Practice: Building a Multi-Stage Workflow with Agent Collaboration

Having examined the principles of workflow orchestration, it's time to translate theory into practice. In this hands-on section, we will design and outline the implementation of a multi-stage workflow. This exercise will demonstrate how several LLM agents, each with a specialized role, can collaborate systematically to accomplish a complex task. We'll focus on a common application: generating a comprehensive report based on a user-defined topic.

Scenario: Automated Report Generation Pipeline

Our goal is to build a system that takes a topic from a user and produces a well-structured, informative report. This process naturally breaks down into several stages, each suitable for a dedicated agent. This division of labor allows each agent to focus on its specific expertise, leading to a more effective overall outcome.

The workflow will involve the following agents:

1. Research Agent: Receives the topic, queries external knowledge sources (e.g., simulated web search, document databases) to gather relevant information.
2. Analysis Agent: Processes the raw information from the Research Agent, extracts pertinent facts, identifies patterns, and structures the key insights.
3. Drafting Agent: Takes the structured insights from the Analysis Agent and composes a coherent first draft of the report.
4. Review Agent: Examines the draft for clarity, accuracy, grammar, and completeness. It can either approve the draft or send it back to the Drafting Agent with specific feedback for revision.

Optionally, a final human review step can be incorporated before the report is considered complete, aligning with the human-in-the-loop principles discussed earlier in this chapter.

Defining the Workflow and Agent Interactions

The collaboration between these agents needs a clearly defined flow. We can visualize this as a directed graph where nodes represent agents (or tasks) and edges represent the flow of information or control.

The diagram illustrates the report generation workflow. Each agent performs a distinct step, passing its output to the next agent in the sequence. Note the feedback loop from the Review Agent to the Drafting Agent, allowing for iterative refinement.

Agent Interface Design

For agents to collaborate effectively, they need well-defined interfaces. Each agent should expect a certain input format and produce a predictable output format. JSON is a common choice for structuring these messages.

For example, the Research Agent might:

• Input: {"topic": "Advancements in Renewable Energy Storage"}
• Output: {"topic": "Advancements in Renewable Energy Storage", "raw_data": [{"source": "url1", "content": "..."}, {"source": "doc2", "content": "..."}], "summary": "Initial overview of findings."}

The Analysis Agent would then take this output as its input and produce something like:

• Input: The output from Research Agent.
• Output: {"topic": "Advancements in Renewable Energy Storage", "structured_insights": [{"point": "Lithium-ion battery improvements...", "details": "..."}, {"point": "Solid-state battery research...", "details": "..."}], "proposed_outline": ["Introduction", "Lithium-ion", "Solid-state", "Conclusion"]}

This structured data exchange is fundamental for smooth transitions between workflow stages.

Implementing the Orchestrator

An orchestrator component is responsible for managing the overall workflow. In a simple sequential workflow like this, the orchestrator calls each agent in turn, passing the output of one agent as the input to the next. For more complex scenarios, you might implement a state machine or use a graph-based execution engine as discussed in "State-Driven and Graph-Based Orchestration Models."

Here's a high-level Python-like structure for a simple orchestrator:

# Assume agent classes (ResearchAgent, AnalysisAgent, etc.) are defined elsewhere.
# Each agent class would have a 'run' method taking a dictionary and returning a dictionary.

class ReportOrchestrator:
    def __init__(self):
        self.research_agent = ResearchAgent()
        self.analysis_agent = AnalysisAgent()
        self.drafting_agent = DraftingAgent()
        self.review_agent = ReviewAgent()
        self.max_revisions = 2 # Prevent infinite loops

    def generate_report(self, topic: str) -> dict:
        print(f"Starting report generation for topic: {topic}")

        # Stage 1: Research
        research_input = {"topic": topic}
        research_output = self.research_agent.run(research_input)
        print("Research complete.")

        if not research_output.get("raw_data"):
            print("Research Agent failed to find data. Aborting.")
            return {"status": "error", "message": "No data found."}

        # Stage 2: Analysis
        analysis_output = self.analysis_agent.run(research_output)
        print("Analysis complete.")

        # Stage 3 & 4: Drafting and Review Loop
        current_draft_input = analysis_output
        final_report = None

        for attempt in range(self.max_revisions + 1):
            print(f"Drafting attempt {attempt + 1}...")
            draft_output = self.drafting_agent.run(current_draft_input)
            print("Drafting complete.")

            print("Reviewing draft...")
            review_input = draft_output # Pass the entire draft context
            review_output = self.review_agent.run(review_input)
            print("Review complete.")

            if review_output.get("status") == "approved":
                final_report = review_output.get("final_report", draft_output.get("draft_text"))
                print("Report approved!")
                break
            elif attempt < self.max_revisions:
                print("Revisions requested. Preparing for redraft.")
                # The review_output should contain feedback for the drafting agent
                # This feedback needs to be incorporated into the input for the next drafting iteration
                current_draft_input = {
                    **analysis_output, # Original insights
                    "previous_draft": draft_output.get("draft_text"),
                    "feedback": review_output.get("feedback")
                }
            else:
                print("Maximum revisions reached. Report not approved.")
                final_report = review_output.get("final_report_as_is", draft_output.get("draft_text")) # Or handle as error
                break
        
        # Stage 5: Optional Human Review (outside this main flow, but noted)
        if final_report:
            print("Report generation process finished.")
            return {"status": "success", "report": final_report}
        else:
            print("Report generation failed after revisions.")
            return {"status": "error", "message": "Failed to produce an approved report."}

# Example usage:
# orchestrator = ReportOrchestrator()
# report_topic = "The Impact of Quantum Computing on Cryptography"
# result = orchestrator.generate_report(report_topic)
# if result["status"] == "success":
#     print("\nFinal Report:\n", result["report"])
# else:
#     print("\nReport Generation Failed:\n", result["message"])

In this pseudo-code:

• Each agent is instantiated.
• The generate_report method executes the sequence.
• Data is passed explicitly.
• A simple revision loop is implemented between the Drafting and Review agents. The ReviewAgent would need to be designed to output a status (approved or needs_revision) and detailed feedback. The DraftingAgent would then need to be able to accept this feedback to improve its next draft.

Agent Implementation Considerations

While we are not detailing the full LLM prompt engineering for each agent here (as that is covered in prerequisite material and earlier chapters on agent design), consider the following for each agent's run method:

• Research Agent: Its LLM prompt would focus on identifying search queries, potentially processing search results (if a tool is used), and summarizing findings. It might also involve deciding when enough information has been gathered.
• Analysis Agent: This agent's prompt would guide the LLM to sift through the researcher's output, identify core themes, discard irrelevant information, and structure the remaining data logically. It might produce a bulleted list of key points or a structured JSON object.
• Drafting Agent: The prompt would instruct the LLM to take the structured insights and expand them into a flowing narrative, adhering to a specified tone or style. It might receive an outline from the Analysis Agent.
• Review Agent: This agent could use a checklist-based prompting strategy. It would ask the LLM to evaluate the draft against criteria like coherence, factual accuracy (if verifiable against provided source snippets), grammatical correctness, and completeness based on the initial structured insights. Its output should clearly state if the draft is approved or requires revision, along with specific, actionable feedback.

Running and Observing the Workflow

To truly build this, you would integrate with your chosen LLM framework. During execution, you would monitor:

• The data objects passed between agents.
• The LLM calls made by each agent.
• The intermediate outputs (e.g., the raw research data, the structured insights, each version of the draft).
• The final report's quality.

Logging (as discussed in Chapter 6) is very important here. Comprehensive logs allow you to trace the flow of data and decisions, which is indispensable for debugging and understanding agent behavior.

Expanding the Workflow: Further Practice

Once this basic pipeline is functional, consider these enhancements as further exercises, drawing from other concepts in this chapter:

1. Error Handling and Resilience: What happens if an agent fails (e.g., an API error, malformed output)? Implement try-catch blocks within the orchestrator or individual agents. How would the system recover or report the failure? (Relates to "Addressing Failures and Ensuring Reliability").
2. Adaptive Branching: Modify the orchestrator to make decisions based on agent outputs. For example, if the Research Agent finds very little information, the orchestrator might trigger a "Topic Refinement Agent" before proceeding to analysis. (Relates to "Adaptive Task Planning").
3. Parallel Execution: Could any parts of this workflow be parallelized? For instance, if the topic has multiple sub-aspects, could multiple Research Agents work in parallel, with their results aggregated before analysis? (Relates to "Managing Resources and Agent Workload Distribution").
4. Human-in-the-Loop for Mid-Workflow Decisions: Introduce a step where a human reviews the structured insights from the Analysis Agent before drafting begins, allowing for course correction. (Relates to "Incorporating Human Oversight").
5. Tool Integration: Equip the Research Agent with a genuine web search tool or document retrieval capability. This involves handling API calls and managing the data returned by external tools. (Relates to Chapter 2's "Integrating External Tools and Functions for Agents").

Takeaways

This exercise in building a multi-stage workflow should solidify your understanding of:

• The necessity of clear role definition: Each agent needs a well-defined purpose and scope.
• The importance of structured inter-agent communication: Standardized data formats are essential for smooth handoffs.
• The function of an orchestrator: Even a simple orchestrator provides essential control over the sequence and flow of tasks.
• Iterative development: Workflows often involve cycles and refinements, like the draft-review loop.
• Scalability and complexity: This relatively simple example provides a foundation upon which more complex collaborations involving many more agents and more intricate dependencies can be built.

By working through the design and (even high-level) implementation of such a system, you are well on your way to mastering the construction of sophisticated multi-agent LLM applications. The patterns and challenges encountered here are representative of those found in more advanced and larger-scale agent teams.