Contract Analytics: Knowledge Management

Building knowledge management artifacts for contracts #

Few would debate the idea that contracts are the most common legal document that most organizations deal with. However, for many organizations, contracting is far from a process-driven or systematic activity. In fact, many organizations have no idea how many contracts they have, where they are, or what they say.

At the opposite end of the spectrum, some organizations have invested heavily in knowledge management and process-driven thinking around their contracts. These organizations have a clear idea of not just how many contracts they have, but also the terms therein and the financial impact of those terms.

Historically, the process of building knowledge management artifacts for contracts has been a manual, time-consuming process. To make matters worse, many of the most important documents for organizations are the most lengthy and complex ones. Only the very largest organizations had the resources to dedicate experienced resources to the documentation and analysis of these documents.

Today, the calculus has changed. With the dramatic improvements in natural language processing demonstrated by the latest generation of large language models, it is now possible to automate much of the process of building knowledge management artifacts for contracts. While human oversight and implementation is still required, the cost and timelines associated with this process have been dramatically reduced.

In this example, we will demonstrate how to use Kelvin NLP to automate the process of building knowledge management artifacts for contracts. We'll review a set of Power Purchase Agreements ("PPA"), extract their sections and definitions, build a model of the "standard" PPA, and then link the sections and definitions to the model. We'll also demonstrate how to use the model to identify deviations from the standard PPA.

Step 1: Retrieve the Power Purchase Agreement sample #

The first step is to retrieve the sample Power Purchase Agreement. In this example, we'll use a sample of 81 Power Purchase Agreements filed in EDGAR. For more information about how to use Kelvin NLP and Kelvin Research to search for and retrieve documents like these from EDGAR, see the EDGAR Kelvin Research example.

Step 2: Extracting the sections #

The next step is to extract the sections from the Power Purchase Agreements. In this example, we'll use one of the Kelvin NLP section extractors to extract the sections from the Power Purchase Agreements. Kelvin NLP currently supports two different engines for section extraction:

• kelvin.nlp.segments.sections.get_sections - a traditional NLP approach to segmenting documents into sections based on the document structure
• kelvin.nlp.extract.llm.sections.SectionExtractor - an LLM-based approach to extract section labels

In this example, we'll use the kelvin.nlp.extract.llm.sections.SectionExtractor to extract the sections from the Power Purchase Agreements. This extractor combines Kelvin NLP's modular support for multiple LLMs with a validation layer to ensure that extracted sections and their positions are accurate and consistent with the document structure.

Creating a SectionExtractor is a two-step process. First, we need to create an JSON-validating LLM engine. Then, we create the SectionExtractor using the LLM engine. The setup steps are shown below:

# imports
from kelvin.nlp.extract.llm.definitions import SectionExtractor
from kelvin.nlp.llm.engines.json_engine import JSONEngine
from kelvin.nlp.llm.engines.openai_engine import OpenAIEngine

# create a gpt-3.5 instance with JSON validation
llm_json = JSONEngine(engine=OpenAIEngine(model="gpt-4"))

# create the section extractor
section_extractor = SectionExtractor(engine=llm_json)

Once we have the SectionExtractor, we can use it like any Kelvin NLP extractors to annotate spans and extract values from text. The code below shows how to extract the sections from a sample Power Purchase Agreement like this 2006 nuclear PPA:

text = '''PALISADES NUCLEAR POWER PLANT

            POWER PURCHASE AGREEMENT

                     BETWEEN

         ENTERGY NUCLEAR PALISADES, LLC

                       AND

            CONSUMERS ENERGY COMPANY

            DATED AS OF JULY 11, 2006

[...]

ARTICLE I: DEFINITIONS

1.1. DEFINED TERMS

     As used in this Agreement, the following terms shall have the following
     meanings:

1.   "ACCREDITED CAPACITY" shall mean Capacity or Replacement Capacity ...

[...]'''


# get the sections
for section in section_extractor.extract_spans(text):
    print(section)

# table of contents
('ARTICLE I: DEFINITIONS', 801, 823)
('ARTICLE II: PURCHASE OF CAPACITY, ENERGY, AND ANCILLARY SERVICES', 1045, 1109)
('ARTICLE III: PAYMENTS', 1856, 1877)
('ARTICLE IV: MAINTENANCE AND OPERATION', 2100, 2137)
('ARTICLE V: METERING, BILLING AND PAYMENT', 2425, 2465)
[...]
('ARTICLE VI: FORCE MAJEURE', 80072, 80097)
('6.1. CONDITIONS OF EXCUSE FROM PERFORMANCE', 156684, 156726)
('6.2. NO TERMINATION; EXTENSION OF TERM', 159400, 159438)
('6.3. ADJUSTMENT PAYMENTS', 159634, 159658)
('ARTICLE VII: EVENTS OF DEFAULT; REMEDIES', 159834, 159874)
('7.1. LIST OF DEFAULT EVENTS', 159876, 159903)
("7.2. SELLER'S SECURITY", 165839, 165861)
("7.3. BUYER'S SECURITY", 168595, 168616)
('7.4. NO CONSEQUENTIAL DAMAGES', 171330, 171359)
[...]

As you can see, the SectionExtractor has extracted the sections from the Power Purchase Agreement and annotated them with their start and end positions in the text. While we aren't interested in the specific numbering of the section headings for later analysis, we can clean up the section headings by removing the numbering and punctuation in later steps.

Step 3: Extracting the definitions #

The next step is to extract the definitions from the Power Purchase Agreements. Just like with the section extraction above, we'll use an LLM-based extractor to extract the definitions from the Power Purchase and organize them for further processing and analysis.

# create the definition extractor with the same engine as above
definition_extractor = DefinitionExtractor(engine=llm_json)

# get the definitions
for definition in definition_extractor.extract_spans(text):
    print(definition)

({'term': 'Accredited Capacity', 'definition': 'Capacity or Replacement Capacity that (a) meets the resource adequacy requirements in Module E of the MISO Tariff, as amended or superseded ("Module E"), and (b) is measured in accordance with the "Criteria and Method For the Uniform Rating of Generating Equipment" set forth in ECAR 4; provided, however, that if either requirement in (a) or (b) is inapplicable, or if both are inapplicable, then Accredited Capacity shall mean Capacity or Replacement Capacity that meets the applicable requirements for Capacity (the "Effective Capacity Requirements") of any Governing Authority having jurisdiction over Buyer, including any Capacity from the Facility that may be deemed available under the Effective Capacity Requirements even if the Facility is not operating.'}, 26752, 26771)
({'term': 'Administrative Committee', 'definition': 'The meaning set forth in Article XII.'}, 108499, 108523)
({'term': 'Affiliate', 'definition': 'With respect to any Person, any other Person (other than an individual) that, directly or indirectly, through one or more intermediaries, controls, or is controlled by, or is under common control with, such Person. For this purpose, "control" means the direct or indirect ownership of fifty percent (50%) or more of the outstanding capital stock or other equity interests having ordinary voting power.'}, 120806, 120815)
({'term': 'Agreement', 'definition': 'This Power Purchase Agreement entered into by Seller and Buyer, including all Exhibits and any and all subsequent modifications or amendments hereto made in accordance herewith.'}, 120894, 120903)
({'term': 'Ancillary Services', 'definition': 'Those services during the Term that are necessary to support the transmission of electric capacity and energy, and support the generation or transmission of Energy from the Facility while maintaining reliable operation of the transmission system, associated with or otherwise corresponding to the Capacity of the Facility and/or output of Energy at such time, which Ancillary Services shall include but not be limited to Reactive Power, regulation, and frequency response service.'}, 127117, 127135)
[...]

Step 4: Inferring model clauses from the corpus #

In our code snippets above, we demonstrated how to extract the sections and definitions from a single Power Purchase Agreement. However, the real value of the LLM-based extractors is that they can be used to extract sections and definitions from a large corpus of Power Purchase Agreements like the one we described in Step 1 above.

In this step, we'll assume that we've extracted all defined terms and section labels from the corpus of PPAs and stored them in the definition_list and section_list variables below.

One simple approach to inferring a standard PPA model from this information is to look for the most common section labels and defined terms in the corpus. These "least common denominator" elements of a PPA can then be used to create a standard PPA model that can be used to extract information from new Power Purchase Agreements. A sample implementation of this approach is shown below.

# import
from collections import Counter

# get the most common section labels
section_counter = Counter(section_list)
print(section_counter.most_common(10))

# get the most common defined terms
definition_counter = Counter(definition_list)
print(definition_counter.most_common(10))

The code above prints the 10 most common section labels and defined terms in the corpus of Power Purchase Agreements. While ten is an arbitrary number, it is a good starting point for identifying the most common elements of a PPA. The output of the code above is shown below.

# clauses
[
  ('FORCE MAJEURE', 50),
  ('NOTICES', 41),
  ('TERM', 39),
  ('SEVERABILITY', 29),
  ('CONFIDENTIALITY', 28),
  ('DEFINITIONS', 22),
  ('INDEMNIFICATION', 21),
  ('ASSIGNMENT', 20),
  ('LIMITATION OF LIABILITY', 20),
  ('DELIVERY POINT', 20)
]

# defined terms
[
  ('EFFECTIVE DATE', 12),
  ('ENERGY', 10),
  ('DELIVERY POINT', 9),
  ('POWER PURCHASE AGREEMENT', 7),
  ('CONTRACT YEAR', 7),
  ('FERC', 7),
  ('ANCILLARY SERVICES', 7),
  ('AGREEMENT', 7),
  ('CAPACITY', 7),
  ('BUSINESS DAY', 6)
]

To keep the example simple, we'll focus our attention on the risk-related provisions that occur. However, to make things more interesting, let's ask an LLM to identify the risk-related provisions for us instead of manually identifying them ourselves.

To do this, we'll use the kelvin.nlp.llm.qa module to ask the LLM to identify the common risk-related provisions from our frequency data. The code below shows how to do this.

# setup the input data based on our frequency data
input_data = "### Frequency of Contract Clauses\n" + json.dumps(section_data.most_common(50))

# setup a QA engine with GPT-4
answerer = TextAnswerer(engine=OpenAIEngine(model="gpt-4"))

# ask the LLM to identify the most common risk-related sections from our data
print(answerer.get_answer(input_data, "List the most important risk-related contract clauses, "
                                          "combining similar clause types when appropriate."))

The LLM's answer to our question is shown below. As you can see, the LLM was able to filter the sections to narrow their focus to important risk-related sections.

1. Force Majeure (54): Force majeure clauses address the occurrence of unforeseen events that are beyond the control of the parties and may lead to delay or inability to perform obligations under the contract.
2. Indemnification (22): Indemnification clauses provide for one party to compensate the other for losses or damages incurred due to the actions or inactions of the indemnifying party.
3. Limitation of Liability (21): Limitation of liability clauses limit the amount of financial responsibility one or both parties may have if found responsible for damages, losses or other legal liabilities.
4. Governing Law (16): Governing law clauses determine the jurisdiction and the set of laws under which the parties agree to have their legal disputes resolved.
5. Representations and Warranties (16): Representations and warranties clauses include the factual statements and assurances made by the parties to the contract, which serve as a basis for entering into the agreement.
6. Compliance (16): Compliance clauses require the parties to adhere to specific laws, regulations, and standards that are relevant to the performance of their contractual obligations.
7. Event of Default (14): Event of default clauses outline the specific circumstances that would constitute a default under the contract, triggering consequences such as termination or other specified remedies.
8. Remedies (13): Remedies clauses specify the actions that may be taken by either party in case of breach or default under the contract.
9. Termination (9): Termination clauses outline the grounds and procedures for ending the contract before the end of its term, typically due to breach or default by one of the parties.

Now that we have a list of the most common risk-related sections, let's demonstrate how we might ask the model to abstract the text of these sections into a standard or model clause. To do this, we'll retrieve the text of all sections from the sample, combine them into a single prompt, and ask the LLM to generate a model clause. The code below shows how to do this.

# setup the engine
llm_gpt4 = engine=OpenAIEngine(model="gpt-4")

input_data = """### Force Majeure Clause Samples\n"""
for section in sample_data:
  input_data += f"<EXAMPLE>{section}</EXAMPLE>\n\n"
  # Check if we've exceeded the token limit.
  # In a real scenario, we could use a recursive split answerer or two-stage abstraction to handle this.
  if llm_gpt4.get_token_count(input_data) > llm_gpt4.get_max_token_count():
    break

  # setup answerer and ask the question
  answerer = TextAnswerer(engine=llm_gpt4)
  print(answerer.get_answer(input_data, "Draft a model clause based on the examples above."))

FORCE MAJEURE

12.1 Definition: An event of "Force Majeure" shall be defined as any
interruption or failure of service, deficiency in the quality or
quantity of service, or any other failure to perform any of the
Party's obligations under this Agreement to the extent such failure
occurs without fault or negligence on the part of that Party and
is caused by factors beyond that Party's reasonable control, which
by the exercise of reasonable diligence that Party is unable to
prevent, avoid, mitigate, or overcome, including but not limited
to: acts of God; labor disturbance; extraordinarily severe
weather conditions; war; riots; requirements, actions, or failures
to act on the part of governmental authorities; fire; and damage
to or breakdown of necessary facilities.

12.2 Excuse of Performance: Neither Party shall be deemed to be in
breach of this Agreement, nor shall any event of default occur,
solely to the extent that such Party's delay or failure in the
performance of its obligations under this Agreement (except for
obligations to pay money) is due to an event of Force Majeure.

12.3 Notice and Procedures: The Party claiming an event of Force
Majeure shall (i) give the other Party reasonably prompt written
notice describing the particulars of the occurrence; (ii) suspend
performance only for the scope and duration required by the event
of Force Majeure; (iii) use its reasonable commercial efforts to
remedy its inability to perform; and (iv) when able to resume
performance of its obligations under this Agreement, give the
other Party written notice to that effect.

As you can see, the LLM was able to generate a simple provision based on the examples provided. We could repeat the process with additional prompting to create a scale of high, medium, and low risk clauses from the perspective of the power seller in this scenario.

# ...
print(answerer.get_answer(input_data, "Draft a model clause based on the "
 "examples above that includes a very broad set of force majeure events."))

The output of this altered prompt is shown below. As you can see, the LLM was able to generate a clause that is much broader in scope than the previous example.

FORCE MAJEURE

1. Definition of Force Majeure

1.1 For the purposes of this Agreement, "Force Majeure" shall be defined as any
event or circumstance, whether foreseeable or unforeseeable, which is beyond
the reasonable control of a Party, and which, by the exercise of reasonable
diligence, that Party is unable to prevent, avoid, mitigate, or overcome,
including, but not limited to:

(a) acts of God, including but not limited to fires, explosions, earthquakes,
hurricanes, tornados, floods, or any other natural disaster;

(b) war, rebellion, insurrection, terrorism, riots, civil unrest, or any
other form of political or social unrest;


(c) strikes, lockouts, labor disputes, or any form of industrial action;

(d) epidemics, pandemics, or other public health crises;

[...]

While simple, this example demonstrates how we can use LLMs to generate sample clauses along a spectrum of risk or other "parameters" of analysis. These sample clauses can be useful for the development of contract risk score cards or playbooks with known acceptable or unacceptable alternatives. The LLM-generated samples can even be used as synthetic data to train other traditional or LLM-based models!

Step 5: Checking a new document against the standard PPA model #

In the next example, we'll demonstrate how to use the techniques in this tutorial to build a complete data model of standard PPA clauses and use it to check a new document against the model. Stay tuned!