Natural Language Processing and Computational Linguistics

Abstract

As an engineering field, research on natural language processing (NLP) is much more constrained by currently available resources and technologies, compared with theoretical work on computational linguistics (CL). In today’s technology-driven society, it is almost impossible to imagine the degree to which computational resources, the capacity of secondary and main storage, and software technologies were restricted when I embarked upon my research career 50 years ago. While these restrictions inevitably shaped my early research into NLP, my subsequent work evolved, according to the significant progress made in associated technologies and related academic fields, particularly CL.Figure 1 shows the research topics in which I have been engaged. My initial NLP research was concerned with a question answering system, which I worked on during my M.Eng and D.Eng degrees. The research focused on reasoning and language understanding, which I soon found was too ambitious and ill-defined. After receiving my D.Eng., I changed my direction of research, and began to be engaged in processing forms of language expressions, with less commitment to language understanding, machine translation (MT), and parsing. However, I returned to research into reasoning and language understanding in the later stage of my career, with clearer definitions of tasks and relevant knowledge, and equipped with access to more advanced supporting technologies.In this article, I begin by briefly describing my views on mutual relationships among disciplines related to CL and NLP, and then move on to discussing my own research.Language is a complex topic to study, infinitely harder than I first imagined when I began to work in the field of NLP.There is a whole discipline on the study of language—namely, linguistics. Linguistics is concerned not only with language per se, but must also deal with how humans model the world.1 The study of semantics, for example, must relate language expressions to their meanings, which reside in the mental models possessed by humans.Apart from linguistics, there are two fields of science that are concerned with language, that is, brain science and psychology. These are concerned with how humans process language. Then, there are two disciplines in which we are involved—namely, CL and NLP.Figure 2 is a schematic view of these research disciplines. Both of the lower disciplines are concerned with processing language, that is, how language is processed in our minds or our brains, and how computer systems should be designed to process language efficiently and effectively.The top discipline, linguistics, on the other hand, is concerned with rules that are followed by languages. That is to say, linguists study language as a system. This schematic view is certainly oversimplified, and there are subject fields in which these disciplines overlap. Psycholinguistics, for example, is a subfield of linguistics which is concerned with how the human mind processes language. A broader definition of CL may include NLP as its subfield.In this article, for the sake of discussion, I adopt narrower definitions of linguistics and CL. In this narrower definition, linguistics is concerned with the rules followed by languages as a system, whereas CL, as a subfield of linguistics, is concerned with the formal or computational description of rules that languages follow.2CL, which focuses on formal/computational description of languages as a system, is expected to bridge broader fields of linguistics with the lower disciplines, which are concerned with processing of language.Given my involvement in NLP, I would like to address the question of whether the narrowly defined CL is relevant to NLP. The simple answer is yes. However, the answer is not so straightforward, and requires us to examine the degree to which the representations used to describe language as a system are relevant to the representations used for processing language.Although my colleagues and I have been engaged in diverse research areas, I pick up only on a subset of these, to illustrate how I view the relationships between NLP and CL. Due to the nature of the article, I ignore technical details and focus instead on the motivation of the research and the lessons which I have learned through research.Background and Motivation. Following the ALPAC report Pierce et al. (1966), research into MT had been largely abandoned by academia, with the exception of a small number of institutes (notably, GETA at Grenoble, France, and Kyoto University, Japan). There were only a handful of commercial MT systems, being used for limited purposes. These commercial systems were legacy systems that had been developed over years and had become complicated collections of ad hoc programs. They had become too convoluted to allow for changes and improvements. To re-initiate MT research in academia, we had to have more systematic and disciplined design methodologies.On the other hand, theoretical linguistics, initiated by Noam Chomsky (Chomsky 1957, 1965) had attracted linguists with a mathematical orientation, who were interested in formal frameworks of describing rules followed by language. Those linguists with interests in formal ways of describing rules were the first generation of computational linguists.Although computational linguists did not necessarily follow the Chomskyan way of thinking, they shared the general view of treating language as a system of rules. They had developed formal ways of describing rules of language and showed that these rules consisted of different layers, such as morphology, syntax, and semantics, and that each layer required different formal frameworks with different computational powers. Their work had also motivated work on how one could process language by computerizing its rules of language. This work constituted the beginning of NLP research, and resulted in the development of parsing algorithms for context-free language, finite-state machines, and so forth.3 It was natural to use this work as the basis for designing the second generation of MT systems, which was initiated by an MT project (MU project, 1082-1986) led by Prof. M. Nagao (Nagao, Tsujii, and Nakamura 1985).Research Contributions. When I began research into MT in the late 1970s, there was a common view largely shared by the community, which had been advocated by the group of GETA, in France. The view was called the transfer approach of MT (Boitet 1987).The transfer approach viewed translation as a process consisting of three phases: analysis, transfer, and generation. According to linguists, a language is a system of rules. The analysis and generation phases were monolingual phases that were concerned with a set of rules for a single language, the analysis phase using the rules of the source language and the generation phase using the rules of the target language. Only the transfer phase was a bilingual phase.Another view shared by the community was an abstraction hierarchy of representation, called the triangle of translation. For example, Figure 3(a)4 shows the hierarchy of representation used in the Eurotra project, with their definition of each level (Figure 3(b)).By climbing up such a hierarchy, the differences among languages would become increasingly small, so that the mapping (i.e., the transfer phase) from one language to another would become as simple as possible. Independently of the target language, the goal of the analysis phase was to climb up the hierarchy, while the aim of the generation phase was to climb down the hierarchy to generate surface expressions in the target language. Both phases are concerned only with rules of single languages.In the extreme view, the top of the hierarchy was taken as the language-independent representation of meaning. Proponents of the interlingual approach claimed that, if the analysis phase reached this level, then no transfer phase would be required. Rather, translation would consist only of the two monolingual phases (i.e., the analysis and generation phases).However, in Tsujii (1986), I claimed, and still maintain, that this was a mistaken view about the nature of translation. In particular, this view assumed that a translation pair (consisting of the source and target sentences) encodes the same “information”. This assumption does not hold, in particular, for a language pair such as Japanese and English, that belong to very different language families. Although a good translation should preserve the information conveyed by the source sentence as much as possible in the target sentence, translation may lose some information or add extra information.5Furthermore, the goal of translation may not be to preserve information but to convey the same pragmatic effects to readers of the translation.More seriously, the abstract level of representation such as Interface Structure6 in Eurotra focused only on the propositional content encoded in language, and tended to abstract away other aspects of information, such as the speaker’s empathy, distinction of old/new information, emphasis, and so on.To climb up the hierarchy led to loss of information in lower levels of representation. In Tsujii (1986), instead of mapping at the abstract level, I proposed “transfer based on a bundle of features of all the levels”, in which the transfer would refer to all levels of representation in the source language to produce a corresponding representation in the target language (Figure 4). Because different levels of representation require different geometrical structures (i.e., different the of this had to for development of a mathematical of representation with which levels (i.e., to be with their mutual relationships the we to the transfer phase was transfer and Tsujii which was by the of in CL. According to the views of linguists at the a language is an set of expressions in is defined by a set of rules. this number of one generate infinitely of the language. claimed that the of a was by the of its using the rules that the translation the same to translation. That is, the translation of a was by the of its In this of infinitely of the source language could be the translation the translation of a sentence would be by a of a source The translation of a would then be by the of its That is, translation would be in a up from of translation to the mapping of a from the source to the target would be by the of the the for the how to a to the In the project, we called this transfer and Tsujii (Figure with the MT systems, which source expressions with target in an and ad hoc the of transfer in the project was defined and and development of the MT systems from research into CL, more defined and design than MT The project and MT systems the of these design we could not have these in such a of the differences between the of the two disciplines also CL to focus on aspects of language as morphology, syntax, semantics, MT systems must be to all aspects of information conveyed by language. As climbing up a hierarchy that focuses on propositional content does not in good more between CL and NLP is the of of is the single significant in NLP it requires the in which expressions to be to be In other it requires understanding of of are in Figure The Japanese a of in and would be into have and so on the for with transfer, it requires to be (i.e., the in which a to be The nature of made the process of transfer was also a in the analysis which I in the general of CL or linguistics is that it to view language as an system and the of understanding, which requires to or However, NLP require an understanding or of language expressions in of and which may other such as and so I this in the on the of research.Background and Motivation. the I was engaged in MT research, in CL, its early in theoretical linguistics by Chomsky assumed that of of rules the two levels of that is, and surface A way of was also shared by the MT They assumed that climbing up the hierarchy would of which from the representation at one level to another representation at the Because each level of the hierarchy required its own geometrical it was not possible to have a representation, in which representations of all the levels view was changed by the of that used to allow of from one level to it mutual relationships among different levels of representation in a This view was in with our of transfer, which used a bundle of features of different levels for some at the the of That is, structures of all the levels are constrained by the of a and these are encoded in This was also in with our significant development in CL at the same a number of the and the had linguistics and to have significant on research into CL and NLP et al. the NLP of view, the of led to the development of (i.e., for research that would these two to the analysis is, parsing based on Contributions. It is claimed that of In the analysis phase of the up the lower levels of processing could not refer to in levels of representation. This was the main of the of at the early of climbing up the analysis could not refer to that in analysis would the other hand, the could describe at all levels in a single it was possible to refer to at all to down the set of possible in the was still This was we not have ways of and pragmatic linguists were interested in formal ways for and levels of representation, but not so much in how are to be To or pragmatic one may have to refer to the mental models of the (i.e., how humans the or structures single and so These of the of CL research at the main focus is on it is whether or be used as They may be more concerned with the of an than the which an should example, the in for parsing using of and how to the of for models were one of the for and the of an However, models for such as and context-free had to be changed for more complex for such as had to be for were in of describing in a the which was a for treating was very To NLP systems, we had to technologies and processing for at the of to study how we could a into and representations for parsing. The research of an abstract machine for processing of structures and development of a et al. et al. into a more representation, such as and Tsujii et al. and from of structures and models and Tsujii of parsing based on of and their and Tsujii et al. representation of our parsing model is in Figure a sentence, its representation of all the levels was at the stage by using the in the first two The first phase was a that would to in a were from the and a set of were to a in its A would the of for the of The was a which could be in a very and Tsujii This that the surface (i.e., of was used for of second phase was A which also was from the was used to whether of by the first phase could a The did not to whether a could or The second phase of would that could not phase not only the representation of all the but it also extra in the Because the first two phases only use in the the phase would by the first two phases if they to these extra In this the system would to the phases to the of these research a based on As in CL were for the systematic development of NLP changed the view of parsing as up the the systematic of of of into and so These also for in NLP such as of and so which are for treating the other hand, of CL to NLP did not this would in had to into more which required significant and on the NLP For example, we had to the into such as and so It is that, while the was to the hierarchy each stage in the was defined and related to each other through the single also that in the fields of computer changed was possible to in NLP. For example, the design of an abstract machine and its for in et al. systems for of and Tsujii Tsujii, and and so would be impossible in the broader fields of computer and much computational et al. the other hand, the in NLP. models in of the with the that we for the the NLP community with systematic ways of with of different models also which made systematic development of NLP systems possible. However, the in parsing still are the in of the number of was the sentence level, the That is, a sentence in which all are very Because of are (i.e., of or of in such as of and Tsujii using there are to a understanding is taken into This to the research language and and Motivation. I was interested in the topic of how to relate language with at the very beginning of my the my led to that a of could be used as a and was engaged in research of a system based on a and Tsujii However, resources such as a of processing of computer systems, and NLP technologies, such as were not available at the soon that the research would a whole of research topics in such as representation of common human ways of and so the topics had to deal with and of or the models that humans have may from one to I that the research target was through research in MT and parsing in the later of my career, I to that NLP research is if it how is in and that NLP are all related to of understanding and the same NLP as an engineering field, I it to be to have a definition of or information with which language is to be I would like to too much of research into and reasoning and to our research focus to the between language and As a research I to focus on the as the There were two for the was that colleagues at the two with which I was that, in to it had become increasingly for to of information in a of in diverse subject fields such as and In to the of they also had diverse that had to be with each In other they had a of shared by that was to be with information in other was that there were colleagues at the of who were interested in According to the on information in a by the group and research into at the of on between and and et al. the had been a natural of The was that information in a and were defined by the target and not by NLP had and to their own in the target Contributions. Although there had been a of research into information and for the there had been no to NLP to in the To address the at the of and the of a research in this this was a research we first had to tasks to to resources, and to not only NLP but also in the target the involvement of NLP and we found that a in the also began to be interested in research In to we a number of research in with colleagues the which led to of a at The first in with the The and shared It been and become one of the in NLP The research field of of NLP to is to et al. et al. and science et al. by the two include the et al. and Tsujii a of with their and Tsujii the et al. the et al. a design for information et al. an system based on Tsujii, and a system for et al. is one of the used in the To information in and how it was encoded in language, we not only from the of view but also from the of of and and and were to the et al. were with of the target which had been by the target to information in diverse in we developed a with which is used by the NLP In with we defined a set of NLP tasks et al. and Tsujii and Tsujii et al. and developed a set of et al. et al. et al. for which were to be into to of of et al. et al. a of this we between such as and and such as and and that (Figure The mapping between structures and the defined by was more complex than the mapping assumed by computational soon that, as an NLP information was very different from In particular, there were differences between the information that the to convey and in and the information that the readers (i.e., a group of to and from of the information that the to the would the information that they were interested of as an NLP made the mapping from language to information very different from the transfer phase in which to the same information in the source and target languages. While the of from structures (i.e., and their structures would only for the of and and these were to be with other The mapping to the transfer based on a bundle of the transfer in the level of representation, we first used the to climb up the hierarchy to the we called from which we to a set of to (Figure assumed that, based on surface of may be this would at a level of is, the same approach to simple transfer at the abstract Although this approach it soon reached its increasingly and we that this was of the nature of and to the approach based on a bundle of features (Figure et al. This to the research, which a language model In this information is assumed to be in the language The information is not in systems (Figure and et al. the other hand, our in the tasks and of In this it that structures more significant example, about an from different each As for the or of are fields such as and about an be made or with different of To the degree of of a we have to examine the of in which the is et al. and on the of an in a sentence, it may be as a and so The in which information by a be to and an research of an is the of in which an NLP system is used to a set of different from different to a of et al. In this a system must be to whether two in different be as the same in a To the system must be to the in which two by the of the This may also require structures to be taken into on the we forms of (i.e., by the target and diverse which include The of information in with these resources to research topics on the between language and of the target Because such as have views on which of information their knowledge, we the and of that may have research in the general with which NLP technologies such as parsing have been less than we research into the of information to the of is the nature of human The CL and NLP to reasoning as a of process based on However, the reasoning that the in the may not be so in example, we found that the reasoning by on is based on between For example, they that a A is to be in an by a that a which is to is in the same of The between A and is based on the between their Because such among are in their in language models on a of would be to their (i.e., a of also to such it may be to use of information, such as of collections of and so to such among and to reasoning based on have the progress and significant changes that models and have to the field of NLP. This is a in which in broader fields of computer up to and the NLP changes by and are and have had a not only on NLP but also and other of It is a the of diverse NLP I it the NLP by NLP with the processing of other information and with and so fields such as and reasoning based on used different and processing They the same basis of and It is much to forms of that NLP in and NLP with are more than we The research of the with which I are on these et al. et al. and the NLP based on language models is increasingly from other research disciplines that the study of language. The nature of and also the way of NLP systems the are very I that the may to by MT could the by computational but and systematic analysis of this to and the of the through analysis of parsing when the of analysis based on by other disciplines, and of NLP systems and the other hand, CL to language as a system or focus on study on aspects of that language in NLP systems, with NLP CL would be to the of their and to a theoretical basis for of NLP is the to NLP and was to the field of NLP by my who was a of the away It is that I could not my and with my career of almost 50 I have research into NLP at institutes Kyoto of the of and I the on of the research and who worked with at these my research could not have in the way that it I their

Topics & Concepts