IDL-Expressions: A Formalism for Representing and Parsing Finite Languages in Natural Language Processing

We propose a formalism for representation of finite languages, referred to as the class of IDL-expressions, which combines concepts that were only considered in isolation in existing formalisms. The suggested applications are in natural language processing, more specifically in surface natural language generation and in machine translation, where a sentence is obtained by first generating a large set of candidate sentences, represented in a compact way, and then by filtering such a set through a parser. We study several formal properties of IDL-expressions and compare this new formalism with more standard ones. We also present a novel parsing algorithm for IDL-expressions and prove a non-trivial upper bound on its time complexity.

💡 Research Summary

The paper introduces a new formalism called IDL‑Expressions for compactly representing finite languages, with a focus on applications in surface natural language generation and machine translation. In hybrid generation systems, a large set of candidate sentences is first produced (often on the order of 10¹²) and then filtered using a parser that exploits target‑language knowledge. Existing compact representations—bags of words, word lattices, and non‑recursive context‑free grammars (CFGs)—each suffer from a specific limitation: bags cannot encode precedence constraints and lead to NP‑complete parsing; word lattices can encode precedence and disjunction but require exponential space to model free word order; non‑recursive CFGs are compact and expressive but also lack a primitive for unrestricted word order, forcing exponential blow‑up for permutations.

IDL‑Expressions combine three operators: I (Interleave), D (Disjunction), and L (Lock). The Interleave operator allows two sub‑expressions to be merged in any order, thereby modeling free word order. Disjunction provides alternative lexical or phrasal choices. The Lock operator restricts Interleave to a bounded region, enforcing a fixed order where necessary (e.g., to respect syntactic dependencies). By nesting these operators, one can simultaneously express “free ordering between phrases” and “fixed ordering inside a phrase,” something none of the earlier formalisms can do without a combinatorial explosion.

The authors translate an IDL‑Expression into an IDL‑graph, a directed acyclic graph whose nodes are tokens and whose edges encode the three operators. A novel notion of a cut captures the set of active parsing states at any point in the graph; each cut corresponds to a particular configuration of Interleave expansions. The number of distinct cuts, denoted #cuts, determines the complexity of parsing.

Parsing is performed by adapting the classic Earley algorithm to work on IDL‑graphs. Earley items retain the form (non‑terminal, start‑position, current‑position) but are now associated with cuts rather than linear positions. Prediction, scanning, and completion steps proceed as usual, except that during an Interleave the parser may simultaneously advance multiple items, reflecting the multiple possible interleavings. The Lock operator forces the algorithm to behave like a standard Earley parser within locked regions, thereby preserving precedence constraints.

A central theoretical contribution is an upper bound on #cuts. By analyzing the structure of the graph—particularly the number k of Lock operators and the size n_i of each locked segment—the authors prove that #cuts ≤ ∏_{i=1}^{k} (n_i + 1), a polynomial bound in the size of the input. Consequently, the overall time complexity of the parsing algorithm is O(|G| · #cuts · n³), where |G| is the size of the context‑free grammar used for parsing and n is the total number of tokens. Space consumption is O(|G| · #cuts · n). This shows that, unlike word lattices, the algorithm remains polynomial even when the underlying language permits unrestricted word order, provided that enough Lock operators are placed to limit interleaving.

The paper also discusses practical implications. IDL‑Expressions can represent candidate sets that would otherwise require exponential space, yet they can be parsed efficiently enough for real‑world MT pipelines. The authors suggest extensions such as attaching probabilistic weights to I, D, and L operators for ranking candidates, integrating semantic graphs as input, and parallelizing the parsing process.

In summary, IDL‑Expressions fill a gap in finite‑language representation by unifying interleaving, disjunction, and order‑locking in a compact graph‑based format. The associated parsing algorithm extends Earley’s dynamic programming approach, achieves polynomial time under realistic constraints, and opens the door to more scalable surface generation and filtering in large‑scale natural language processing systems.