2.1 Knowledge Authoring Logic Machine for factual language

The Knowledge Authoring Logic Machine (KALM) (Gao et al. Reference Gao, Fodor and Kifer2018a;Reference Gao, Fodor and Kiferb) allows users to author knowledge using Attempto Controlled English (ACE) (Fuchs and Schwitter Reference Fuchs and Schwitter1996). However, ACE’s grammar is too limiting and poses a high learning curve, particularly for non-technical users. To mitigate this problem, KALM was extended to KALM for Factual (English) Language (KALM^FL) (Wang et al. Reference Wang, Borca-Tasciuc, Goel, Fodor and Kifer2022) by introducing factual (English) sentences and focusing on authoring facts and simple queries. Factual sentences express atomic database facts and queries (e.g. “Mary buys a car”). They can become more complex with adnominal clauses (e.g. “Mary buys a car that is old”) and can be combined via “and” and “or” (e.g. “Mary buys a car and Bob buys a watch”). In comparison, sentences not expressing factual information (e.g. “Fetch the ball” or “Oh, well”) are non-factual and are not allowed. Factual sentences can be captured through properties based on dependency analysis and Part-of-Speech (POS) tagging (Wang et al. Reference Wang, Borca-Tasciuc, Goel, Fodor and Kifer2022), which is a very mild restriction compared to complex grammars such as in ACE. This means that users do not need to master complex grammars. Instead, they can simply write normal sentences that describe database facts or basic Boolean combinations of facts, and, as long as they avoid fancy language forms, their sentences will be accepted.

KALM^FL is a two-stage system following the structured machine learning paradigm. In the first stage, known as the training stage, KALM^FL constructs logical valence patterns (LVPs) by learning from training sentences. An LVP is a specification that tells how to extract role fillers for the concepts represented by the English sentences related to that LVP. In the second stage, known as the deployment stage, the system does semantic parsing by applying the constructed LVPs to convert factual English sentences into unique logical representations (ULRs). Figure 1(a) depicts the training stage of KALM^FL, with the key steps explained in the accompanying text.

Fig. 1. The frameworks of the KALM^FL system.

Annotating Training Sentences. To enable semantic understanding of a domain of discourse, knowledge engineers must first construct the required background knowledge in the form of KALM^FL frames. The overall structure of most of these frames can be adopted from FrameNet (Baker et al. Reference Baker, Fillmore and Lowe1998) and converted into the logic form required by KALM^FL. Then, knowledge engineers compose training sentences and annotate them using KALM^FL frames. For example, the annotated training sentence (1), below, indicates that the meaning of “Mary buys a car” is captured by the Commerce_buy frame; the word that triggers this frame, a.k.a. the lexical unit (LU), is the 2nd word “buy” or its synonym “purchase”; and, the 1st and the 4th words, “mary” and “car,” play the roles of Buyer and Goods in the frame.

(1)

Syntactic Parsing. KALM^FL then performs syntactic parsing using mStanza ^{Footnote 4} (Wang et al. Reference Wang, Borca-Tasciuc, Goel, Fodor and Kifer2022) and automatically corrects some parsing errors. Figure 2 shows two mStanza parses, where the colored boxes contain POS tags and the labeled arrows display dependency relations.

Fig. 2. mStanza parses.

Construction of LVPs. mStanza parses, along with annotations of sentences, allow KALM^FL to construct LVPs that specify how to fill the roles of a frame triggered by an LU. For example, by synthesizing the information in training sentence (1) and the mStanza parse in Figure 2(a), KALM^FL learns that, to fill the roles Buyer and Goods of the Commerce_buy frame triggered by the LU “buys,” one should extract the subject and object of “buys” through the dependency relations nsubj and obj, respectively. This learned knowledge about role-filling is encoded as an LVP (2) as follows:

(2)

The deployment stage of KALM^FL is illustrated in Figure 1(b). Two key steps in this stage are further explained below.

Frame-based Parsing. When an unseen factual sentence comes in, KALM^FL triggers all possible LVPs using the words in the sentence. Then, the triggered LVPs are applied to this sentence to extract role fillers and a frame-based parse of this sentence is generated. For example, a new sentence “Bob bought a watch” is parsed as Figure 2(b) by mStanza. It triggers LVP (2) using the LU “bought” (whose base form is “buy”). KALM^FL then extracts the role filler “Bob” for the role Buyer according to the dependency list [nsubj]. Similarly, “watch” is extracted for the role Goods according to the dependency [obj].

Constructing ULRs. Ultimately, frame-based parses are represented as ULR facts that capture the meaning of the original English sentences and are suitable for querying. For example, the ULR for the factual sentence “Bob bought a watch,” below, indicates that the meaning of the sentence is captured by the Commerce_buy frame, that “Bob” is the Buyer, and “watch” plays the role of Goods, where rl/2 represents instances of (role, role-filler) pairs.

2.2 Disjunctive information and frame reasoning

Our reasoning subsystem combines Answer Set Programming (ASP) with aspects of frame-based reasoning.

DLV (Leone et al. Reference Leone, Pfeifer, Faber, Eiter, Gottlob, Perri and Scarcello2006) is a disjunctive version of Datalog that operates under the ASP paradigm. It extends Datalog by adding support for disjunction in facts and rule heads, thus providing greater expressiveness for disjunctive information than KRR systems based on the well-founded semantics (e.g. XSB Swift and Warren Reference Swift and Warren2012). Furthermore, DLV’s support for function symbols and querying makes it more convenient for working with frames (Fillmore et al. 2006) than other ASP systems, such as Potassco (Gebser et al. Reference Gebser, Kaminski, Kaufmann and Schaub2019).

F-logic (Kifer et al. Reference Kifer, Lausen and Wu1995; Kifer and Lausen Reference Kifer and Lausen1989) is a knowledge representation and ontology language that combines the benefits of conceptual modeling with object-oriented and frame-based languages. One of its key features is the ability to use composite frames to reduce long conjunctions of roles into more compact forms, matching ideally the structure of FrameBase’s frames. For example, F-logic frames ^{Footnote 5} can be used to answer the question “What did Mary buy?” given the fact “Mary bought a car for Bob,” whose ULRs, shown below, are not logically equivalent (the fact has more roles than the query).

2.3 Event Calculus for reasoning about actions and their effects

The event calculus (EC) (Kowalski and Sergot Reference Kowalski and Sergot1989) is a set of logical axioms that describe the law of inertia for actions. This law states that time-dependent facts, fluents, that are not explicitly changed by an action preserve their true/false status in the state produced by that action. Here we use the simplified event calculus (SEC) (Sadri and Kowalski Reference Sadri and Kowalski1995), which is a simpler and more tractable variant of the original EC. A fluent in SEC is said to hold at a particular timestamp if it is initiated by an action and not terminated subsequently. This is formalized by these DLV rules:

Here happensAt/2 represents a momentary occurrence of action A at a timestamp. If an action is exogenous insertion of a fluent f at time t then we also represent it as happensAt(f,t). Example 1 demonstrates the use of happensAt/2.

Example 1 The sentence “Mary goes to the bedroom. The bedroom is north of the garden.” is represented as follows:

The first happensAt/2 introduces an action of traveling from place to place while the second happensAt/2 uses an observed (i.e. exogenously inserted) fluent “North_of“(bedroom, garden). Observable fluents are supposed to be disjoint from action fluents, and we will use a special predicate, observable/1, to recognize them in SEC rules. Timestamps indicate the temporal relation between the action and the observed fluent. Predicates person/1, place/2, entity/2 define the domain of roles, while timestamp/1 restricts the domain of timestamps.

The predicates initiates(Action, Fluent) and terminates(Action, Fluent) in SEC are typically used to specify domain-specific axioms that capture the initiation and termination of fluents.

3.1 Authoring and reasoning with KALM^RA Rules

Rules are important to KRR systems because they enable multi-step logical inferences needed for real-world tasks, such as diagnosis, planning, and decision-making. Here we address the problem of rule authoring.

3.1.1 Enhancements for representation of facts

First we discuss the representation of disjunction, conjunction, negation, and coreference, which is not covered in KALM^FL.

Conjunction and Disjunction. The KALM^RA system prohibits the use of a mixture of conjunction and disjunction within a single factual sentence to prevent ambiguous expressions such as “Mary wants to have a sandwich or a salad and a drink.” To represent a factual sentence with homogeneous conjunction or disjunction, the system first parses the sentence into a set of component ULRs. For conjunction, KALM^RA uses this set of ULRs as the final representation. For disjunction, the component ULRs are assembled into a single disjunctive ULR using DLV’s disjunction v as shown in Example 2.

Example 2 The factual sentence with conjunction “Daniel administers a parenteral and an oral antimicrobial therapy for Mary” is represented as the following set of ULRs:

where the predicates doctor, patient, therapy, and method define the domains for the roles. These domain predicates will be omitted in the rest of the paper, for brevity.

The disjunctive factual sentence “Daniel administers a parenteral or an oral antimicrobial therapy for Mary” is represented as the following ULR:

Negation. The KALM^RA system supports explicit negation through the use of the negative words “not” and “no.” Such sentences are captured by appending the suffix “_not” to the name of the frame triggered by this sentence.

Example 3 The explicitly negated factual sentence “Daniel’s patient Mary does not have UTI” is represented by

Coreference. Coreference occurs when a word or a phrase refers to something that is mentioned earlier in the text. Without coreference resolution, one gets unresolved references to unknown entities in ULRs. To address this issue, KALM^RA uses a coreference resolution tool neuralcoref, ^{Footnote 6} which identifies and replaces coreferences with the corresponding entities from the preceding text.

Example 4 The factual sentences “Daniel’s patient Mary has UTI. He administers an antimicrobial therapy for her.” are turned into

where the second ULR uses entities “Daniel” and “Mary” instead of the pronouns “he” and “she.”

3.1.2 Rule representation

Rules in KALM^RA are expressed in a much more restricted syntax compared to facts since, for knowledge authoring purposes, humans have little difficulty learning and complying with the restrictions. Moreover, since variables play such a key role in rules, complex coreferences must be specified unambiguously. All this makes writing rules in a natural language into a very cumbersome, error-prone, and ambiguity-prone task compared to the restricted syntax below.

Definition 1 A rule in KALM^RA is an if-then statement of the form “If ${P_1}$ , ${P_2}$ , …, and ${P_n}$ , then ${C_1}$ , ${C_2}$ , …, or ${C_m},$ ” where

1. 1. each ${P_i}$ ( $i = 1..n$ ) is a factual sentence without disjunction;

2. 2. each ${C_j}$ ( $j = 1..m$ ) is a factual sentence without conjunction;

3. 3. variables in ${C_j}$ ( $j = 1..m$ ) must use the explicitly typed syntax (Gao et al. 2018a) and must appear in at least one of the ${P_i}$ ( $i = 1..n$ ). For example, in the rule “If Mary goes to the hospital, then $doctor sees Mary,” the explicitly typed variable $doctor appears in the conclusion without appearing in the premise, which is prohibited. Instead, the rule author must provide some information about the doctor in a rule premise (e.g. “and she has an appointment with $doctor”). This corresponds to the well-known “rule safety” rule in logic programming.

4. 4. variables that refer to the same thing must have the same name. For example, in the rule “If $patient is sick, then $patient goes to see a doctor,” the two $patient variables are intended to refer to the same person and thus have the same name.

Here are some examples of rules in KALM^RA.

Example 5 The KALM^RA rule “If $doctor’s $patient is a young child and has an unexplained fever, then $doctor assesses $patient’s degree of toxicity or dehydration” is represented as follows:

KALM^RA supports two types of negation in rules: explicit negation (Gelfond and Lifschitz Reference Gelfond and Lifschitz1991) and negation as failure (with the stable model semantics Gelfond and Lifschitz Reference Gelfond and Lifschitz1988). The former allows users to specify explicitly known negative factual information while the latter lets one derive negative information from the lack of positive information. Explicit negation in rules is handled the same way as in fact representation. Negation as failure must be indicated by the rule author through the idiom “not provable,” which is then converted into the predicate not/1. The idiom “not provable” is prohibited in rule heads.

Example 6 The KALM^RA rule “If not provable $doctor does not administer $therapy for $patient, then $patient undergoes $therapy from $doctor’ is represented as follows:

where patient/1, doctor/1, and therapy/1 are domain predicates that ensure that variables that appear under negation have well-defined domains.

3.1.3 Queries and answers

Queries in KALM^RA must be in factual English and end with a question mark. KALM^RA translates both Wh-variables and explicitly typed variables into the corresponding DLV variables. Example 7 shows how KALM^RA represents a query with variables.

Example 7 The query “Who undergoes $therapy?” has the following ULR:

KALM^RA then invokes the DLV reasoner to compute query answers. DLV has two inference modes: brave reasoning and cautious reasoning. In brave reasoning, a query returns answers that are true in at least one model of the program and cautious reasoning returns the answers that are true in all models. Users are free to choose either mode.

Example 8 For instance, if the underlying information contains only this single fact

In case of “Mary or Bob undergoes a mental therapy,” two models are computed:

In the cautious mode, there would be no answers while the brave mode yields two:

3.2 Authoring and reasoning with actions

Time-independent facts and rules discussed earlier are knowledge that persists over time. In contrast, actions are momentary occurrences of events that change the underlying knowledge, so actions are associated with timestamps. Dealing with actions and their effects, also known as fluents, requires an understanding of the passage of time. KALM^RA allows users to state actions using factual English and then formalizes actions as temporal database facts using SEC discussed Section 2.3. The following discussion of authoring and reasoning with actions will be in the SEC framework.

Reasoning based on SEC requires the knowledge of fluent initiation and termination. This information is part of the commonsense and domain knowledge supplied by knowledge engineers and domain experts via high-level fluent initiation and termination statements (Definition 2) and KALM^RA translates them into facts and rules that involve the predicates initiates/2 and terminates/2 used by Event Calculus. Knowledge engineers supply the commonsense part of these statements and domain experts supply the domain-specific part.

Example 9 shows how KALM^RA represents fluent initiation and termination.

Example 9 The commonsense initiation statement “$person travels to $place initiates $person is located in $place” would be created by a knowledge engineer and translated by KALM^RA as the following rule:

Here, person/1 and place/1 are used to guarantee rule safety. Since, any object can be in one place only at any given time, we have a commonsense termination statement “$person travels to $place1 terminates $person is located in $place2.” This statement would also be created by knowledge engineers and translated by KALM^RA as follows:

KALM^RA also enhances rules by incorporating temporal information, allowing the inference of new knowledge under the SEC framework. The process begins by requiring users to specify their domain knowledge on fluents in the form of rules described in Definition 1. Then KALM^RA translates these rules into ULRs, with each premise and conclusion linked to a timestamp via the holdsAt/2 predicate. We call these rules time-related because they enable reasoning with fluents containing temporal information. Here is an example of a time-related rule.

where all holdsAt/2 terms share the same timestamp T, since the disjunction of conclusion ULRs ULRC1, …, ULRCm holds immediately if all premise ULRs ULRP1, …, ULRPn hold simultaneously at T.

KALM^RA incorporates temporal information in queries also using holdsAt/2. In this representation, the second argument of holdsAt/2 is set to the highest value in the temporal domain extracted from the narrative. For Example 1, a time-related query can be represented as holdsAt(ULRQ,3)?, where ULRQ is the ULR of the query and 3 is the timestamp that exceeds all the explicitly given timestamps.

4.1 Evaluation of rule authoring

The UTI guidelines (Committee on Quality Improvement 1999) are a set of therapeutic recommendations for the initial Urinary Tract Infection (UTI) in febrile infants and young children. The original version in English was rewritten into the ACE CNL (Shiffman et al. Reference Shiffman, Michel, Krauthammer, Fuchs, Kaljurand and Kuhn2009) for the assessment of ACE’s expressiveness. We rewrite the original English version into factual English, as shown in Appendix A. This new version has a significant number of rules with disjunctive heads, as is common in the real-world medical domain.

The experimental results show that KALM^RA is able to convert the UTI guidelines document into ULRs with 100% accuracy.

4.2 Evaluation of authoring of actions

The 20 bAbI tasks (Weston et al. Reference Weston, Bordes, Chopra, Rush, van Merriënboer, Joulin and Mikolov2015) were designed to evaluate a system’s capacity for natural language understanding, especially when it comes to actions. They cover a range of aspects, such as moving objects (tasks 1-6), positional reasoning (task 17), and path finding (task 19). Each task provides a set of training and test data, where each data point consists of a textual narrative, a question about the narrative, and the correct answer. Figure B.1 in Appendix B presents 20 data points from 20 bAbI tasks, respectively. We used the test data for evaluation. Each task in the test data has 1000 data points.

The comparison systems in this evaluation include a state-of-the-art neural model on bAbI Tasks, STM (Le et al. Reference Le, Tran and Venkatesh2020); an approach based on inductive learning and logic programming (Mitra and Baral Reference Mitra and Baral2016) that we call LPA here; and a recent sensation, ChatGPT. The comparison with STM and ILA results is displayed in Table 1, where “#I&T” denotes the number of user-given initiation and termination statements (Definition 2) used to specify each particular task in KALM^RA. The table shows that KALM^RA achieves accuracy comparable to STM and ILA. ChatGPT has shown impressive ability to give correct answers for some manually entered bAbI tasks even though (we assume) it was not trained on that data set. However, it quickly became clear that it has no robust semantic model behind its impressive performance and it makes many mistakes on bAbI Tasks. The recent (Jan 30, 2023) update of ChatGPT fixed some of the cases, while still not being able to handle slight perturbations of those cases. Three such errors are shown in Table 2, which highlights the need for authoring approaches, like KALM^RA, which are based on robust semantic models.

Table 1. Result comparisons.

Table 2. ChatGPT error cases.

As to KALM^RA, it does not achieve 100% correctness on Tasks 13 (Compound Coreference) and 16 (Basic Induction). In Task 13, the quality of KALM^RA’s coreference resolution is entirely dependent on the output of neuralcoref, the coreference resolver we used. As this technology improves, so will KALM^RA. Task 16 requires the use of the induction principles adopted by bAbI tasks, some of which are questionable. For instance, in Case 2 of Table 3, the color is determined by the maximum frequency of that type, whereas in Case 3, the latest evidence determines the color. Both of these principles are too simplistic and, worse, contradict each other.

Table 3. KALM^RA error cases.