2022
Ferrer-i-Cancho, R.; Bentz, C.; Seguin, C.
Optimal coding and the origins of Zipfian laws Journal Article
In: Journal of Quantitative Linguistics, vol. 29, no. 2, pp. 165-194, 2022.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law for word frequencies, Zipf's law of abbreviation
@article{Ferrer2019c,
title = {Optimal coding and the origins of Zipfian laws},
author = {R. Ferrer-i-Cancho and C. Bentz and C. Seguin},
url = {https://arxiv.org/abs/1906.01545},
doi = {10.1080/09296174.2020.1778387},
year = {2022},
date = {2022-01-01},
journal = {Journal of Quantitative Linguistics},
volume = {29},
number = {2},
pages = {165-194},
abstract = {The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding – under an arbitrary coding scheme – and show that it predicts Zipf's law of abbreviation, namely a tendency in natural languages for more frequent words to be shorter. We apply this result to investigate optimal coding also under so-called non-singular coding, a scheme where unique segmentation is not warranted but codes stand for a distinct number. Optimal non-singular coding predicts that the length of a word should grow approximately as the logarithm of its frequency rank, which is again consistent with Zipf's law of abbreviation. Optimal non-singular coding in combination with the maximum entropy principle also predicts Zipf's rank-frequency distribution. Furthermore, our findings on optimal non-singular coding challenge common beliefs about random typing. It turns out that random typing is in fact an optimal coding process, in stark contrast with the common assumption that it is detached from cost cutting considerations. Finally, we discuss the implications of optimal coding for the construction of a compact theory of Zipfian laws and other linguistic laws.},
keywords = {information theory, Zipf's law for word frequencies, Zipf's law of abbreviation},
pubstate = {published},
tppubtype = {article}
}
The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding – under an arbitrary coding scheme – and show that it predicts Zipf's law of abbreviation, namely a tendency in natural languages for more frequent words to be shorter. We apply this result to investigate optimal coding also under so-called non-singular coding, a scheme where unique segmentation is not warranted but codes stand for a distinct number. Optimal non-singular coding predicts that the length of a word should grow approximately as the logarithm of its frequency rank, which is again consistent with Zipf's law of abbreviation. Optimal non-singular coding in combination with the maximum entropy principle also predicts Zipf's rank-frequency distribution. Furthermore, our findings on optimal non-singular coding challenge common beliefs about random typing. It turns out that random typing is in fact an optimal coding process, in stark contrast with the common assumption that it is detached from cost cutting considerations. Finally, we discuss the implications of optimal coding for the construction of a compact theory of Zipfian laws and other linguistic laws.
2021
Carrera-Casado, David; Ferrer-i-Cancho, Ramon
The advent and fall of a vocabulary learning bias from communicative efficiency Journal Article
In: Biosemiotics, vol. 14, no. 2, pp. 345-375, 2021.
Abstract | Links | BibTeX | Tags: child language, information theory
@article{Carrera2021a,
title = {The advent and fall of a vocabulary learning bias from communicative efficiency},
author = {David Carrera-Casado and Ramon Ferrer-i-Cancho},
url = {https://arxiv.org/abs/2105.11519},
doi = {10.1007/s12304-021-09452-w},
year = {2021},
date = {2021-01-01},
journal = {Biosemiotics},
volume = {14},
number = {2},
pages = {345-375},
abstract = {It is well-known that, when sufficiently young children encounter a new word, they tend to attach it to a meaning that does not have a word yet in their lexicon. In previous research, the strategy was shown to be optimal from an information theoretic standpoint. However, the information theoretic model employed neither explains the weakening of that vocabulary learning bias in older children or polylinguals nor reproduces Zipf's meaning-frequency law, namely the non-linear relationship between the number of meanings of a word and its frequency. Here we consider a generalization of the model that is channeled to reproduce that law. The analysis of the new model reveals regions of the phase space where the bias disappears consistently with the weakening or loss of the bias in older children or polylinguals. In the deep learning era, the model is a transparent low-dimensional tool for future experimental research and illustrates the predictive power of a theoretical framework originally designed to shed light on the origins of Zipf's rank-frequency law.},
keywords = {child language, information theory},
pubstate = {published},
tppubtype = {article}
}
It is well-known that, when sufficiently young children encounter a new word, they tend to attach it to a meaning that does not have a word yet in their lexicon. In previous research, the strategy was shown to be optimal from an information theoretic standpoint. However, the information theoretic model employed neither explains the weakening of that vocabulary learning bias in older children or polylinguals nor reproduces Zipf's meaning-frequency law, namely the non-linear relationship between the number of meanings of a word and its frequency. Here we consider a generalization of the model that is channeled to reproduce that law. The analysis of the new model reveals regions of the phase space where the bias disappears consistently with the weakening or loss of the bias in older children or polylinguals. In the deep learning era, the model is a transparent low-dimensional tool for future experimental research and illustrates the predictive power of a theoretical framework originally designed to shed light on the origins of Zipf's rank-frequency law.
2018
Ferrer-i-Cancho, Ramon; Bentz, Chris
The evolution of optimized language in the light of standard information theory Proceedings Article
In: Cuskley, C.; Flaherty, M.; Little, H.; McCrohon, Luke; Ravignani, A.; Verhoef, T. (Ed.): The Evolution of Language: Proceedings of the 12th International Conference (EVOLANGXII), NCU Press, 2018.
Links | BibTeX | Tags: information theory
@inproceedings{Ferrer2018b,
title = {The evolution of optimized language in the light of standard information theory},
author = {Ramon Ferrer-i-Cancho and Chris Bentz},
editor = {C. Cuskley and M. Flaherty and H. Little and Luke McCrohon and A. Ravignani and T. Verhoef},
url = {http://evolang.org/torun/proceedings/papertemplate.html?p=142},
doi = {10.12775/3991-1.029},
year = {2018},
date = {2018-01-01},
booktitle = {The Evolution of Language: Proceedings of the 12th International Conference (EVOLANGXII)},
publisher = {NCU Press},
keywords = {information theory},
pubstate = {published},
tppubtype = {inproceedings}
}
Ferrer-i-Cancho, R.; Vitevitch, M.
The origins of Zipf's meaning-frequency law Journal Article
In: Journal of the American Association for Information Science and Technology, vol. 69, no. 11, pp. 1369–1379, 2018.
Abstract | Links | BibTeX | Tags: information theory, network science, Zipf's meaning-frequency law
@article{Ferrer2017b,
title = {The origins of Zipf's meaning-frequency law},
author = {R. Ferrer-i-Cancho and M. Vitevitch},
doi = {10.1002/jasist.24057},
year = {2018},
date = {2018-01-01},
journal = {Journal of the American Association for Information Science and Technology},
volume = {69},
number = {11},
pages = {1369–1379},
abstract = {In his pioneering research, G.K. Zipf observed that more frequent words tend to have more meanings, and showed that the number of meanings of a word grows as the square root of its frequency. He derived this relationship from two assumptions: that words follow Zipf's law for word frequencies (a power law dependency between frequency and rank) and Zipf's law of meaning distribution (a power law dependency between number of meanings and rank). Here we show that a single assumption on the joint probability of a word and a meaning suffices to infer Zipf's meaning‐frequency law or relaxed versions. Interestingly, this assumption can be justified as the outcome of a biased random walk in the process of mental exploration.},
keywords = {information theory, network science, Zipf's meaning-frequency law},
pubstate = {published},
tppubtype = {article}
}
In his pioneering research, G.K. Zipf observed that more frequent words tend to have more meanings, and showed that the number of meanings of a word grows as the square root of its frequency. He derived this relationship from two assumptions: that words follow Zipf's law for word frequencies (a power law dependency between frequency and rank) and Zipf's law of meaning distribution (a power law dependency between number of meanings and rank). Here we show that a single assumption on the joint probability of a word and a meaning suffices to infer Zipf's meaning‐frequency law or relaxed versions. Interestingly, this assumption can be justified as the outcome of a biased random walk in the process of mental exploration.
Ferrer-i-Cancho, R.
Optimization models of natural communication Journal Article
In: Journal of Quantitative Linguistics, vol. 25, no. 3, pp. 207-237, 2018.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law for word frequencies
@article{Ferrer2015b,
title = {Optimization models of natural communication},
author = {R. Ferrer-i-Cancho},
doi = {10.1080/09296174.2017.1366095},
year = {2018},
date = {2018-01-01},
journal = {Journal of Quantitative Linguistics},
volume = {25},
number = {3},
pages = {207-237},
abstract = {A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf’s law for word frequencies. The family is a based on a combination of two information theoretic principles: maximization of mutual information between forms and meanings and minimization of form entropy. The family also sheds light on the origins of three other patterns: the principle of contrast; a related vocabulary learning bias; and the meaning-frequency law. Here two important components of the family, namely the information theoretic principles and the energy function that combines them linearly, are reviewed from the perspective of psycholinguistics, language learning, information theory and synergetic linguistics. The minimization of this linear function is linked to the problem of compression of standard information theory and might be tuned by self-organization.},
keywords = {information theory, Zipf's law for word frequencies},
pubstate = {published},
tppubtype = {article}
}
A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf’s law for word frequencies. The family is a based on a combination of two information theoretic principles: maximization of mutual information between forms and meanings and minimization of form entropy. The family also sheds light on the origins of three other patterns: the principle of contrast; a related vocabulary learning bias; and the meaning-frequency law. Here two important components of the family, namely the information theoretic principles and the energy function that combines them linearly, are reviewed from the perspective of psycholinguistics, language learning, information theory and synergetic linguistics. The minimization of this linear function is linked to the problem of compression of standard information theory and might be tuned by self-organization.
2017
Bentz, C.; Alikaniotis, D.; Cysouw, M.; Ferrer-i-Cancho, R.
The entropy of words - Learnability and expressivity across more than 1000 languages Journal Article
In: Entropy, vol. 19, no. 6, 2017.
Abstract | Links | BibTeX | Tags: information theory
@article{Bentz2017a,
title = {The entropy of words - Learnability and expressivity across more than 1000 languages},
author = {C. Bentz and D. Alikaniotis and M. Cysouw and R. Ferrer-i-Cancho},
doi = {10.3390/e19060275},
year = {2017},
date = {2017-01-01},
journal = {Entropy},
volume = {19},
number = {6},
abstract = {The choice associated with words is a fundamental property of natural languages. It lies at the heart of quantitative linguistics, computational linguistics and language sciences more generally. Information theory gives us tools at hand to measure precisely the average amount of choice associated with words: the word entropy. Here, we use three parallel corpora, encompassing ca. 450 million words in 1916 texts and 1259 languages, to tackle some of the major conceptual and practical problems of word entropy estimation: dependence on text size, register, style and estimation method, as well as non-independence of words in co-text. We present two main findings: Firstly, word entropies display relatively narrow, unimodal distributions. There is no language in our sample with a unigram entropy of less than six bits/word. We argue that this is in line with information-theoretic models of communication. Languages are held in a narrow range by two fundamental pressures: word learnability and word expressivity, with a potential bias towards expressivity. Secondly, there is a strong linear relationship between unigram entropies and entropy rates. The entropy difference between words with and without co-textual information is narrowly distributed around ca. three bits/word. In other words, knowing the preceding text reduces the uncertainty of words by roughly the same amount across languages of the world.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
The choice associated with words is a fundamental property of natural languages. It lies at the heart of quantitative linguistics, computational linguistics and language sciences more generally. Information theory gives us tools at hand to measure precisely the average amount of choice associated with words: the word entropy. Here, we use three parallel corpora, encompassing ca. 450 million words in 1916 texts and 1259 languages, to tackle some of the major conceptual and practical problems of word entropy estimation: dependence on text size, register, style and estimation method, as well as non-independence of words in co-text. We present two main findings: Firstly, word entropies display relatively narrow, unimodal distributions. There is no language in our sample with a unigram entropy of less than six bits/word. We argue that this is in line with information-theoretic models of communication. Languages are held in a narrow range by two fundamental pressures: word learnability and word expressivity, with a potential bias towards expressivity. Secondly, there is a strong linear relationship between unigram entropies and entropy rates. The entropy difference between words with and without co-textual information is narrowly distributed around ca. three bits/word. In other words, knowing the preceding text reduces the uncertainty of words by roughly the same amount across languages of the world.
Ferrer-i-Cancho, R.
The placement of the head that maximizes predictability. An information theoretic approach Journal Article
In: Glottometrics, vol. 39, pp. 38-71, 2017.
Abstract | Links | BibTeX | Tags: information theory, word order
@article{Ferrer2013f,
title = {The placement of the head that maximizes predictability. An information theoretic approach},
author = {R. Ferrer-i-Cancho},
url = {Ihttp://hdl.handle.net/2117/108830},
year = {2017},
date = {2017-01-01},
journal = {Glottometrics},
volume = {39},
pages = {38-71},
abstract = {The minimization of the length of syntactic dependencies is a well-established principle of word order and the basis of a mathematical theory of word order. Here we complete that theory from the perspective of information theory, adding a competing word order principle: the maximization of predictability of a target element. These two principles are in conflict: to maximize the predictability of the head, the head should appear last, which maximizes the costs with respect to dependency length minimization. The implications of such a broad theoretical framework to understand the optimality, diversity and evolution of the six possible orderings of subject, object and verb, are reviewed.},
keywords = {information theory, word order},
pubstate = {published},
tppubtype = {article}
}
The minimization of the length of syntactic dependencies is a well-established principle of word order and the basis of a mathematical theory of word order. Here we complete that theory from the perspective of information theory, adding a competing word order principle: the maximization of predictability of a target element. These two principles are in conflict: to maximize the predictability of the head, the head should appear last, which maximizes the costs with respect to dependency length minimization. The implications of such a broad theoretical framework to understand the optimality, diversity and evolution of the six possible orderings of subject, object and verb, are reviewed.
2016
Lozano, A.; Casas, B.; Bentz, C.; Ferrer-i-Cancho, R.
Fast calculation of entropy with Zhang's estimator Book Section
In: Kelih, J. Macutek R. Knight E.; Wilson, A. (Ed.): Issues in Quantitative Linguistics 4. Dedicated to Reinhard Köhler on the occasion of his 65th birthday, pp. 273-285, RAM-Verlag, Lüdenscheid, 2016, (No. 23 of the series ``Studies in Quantitative Linguistic'').
Abstract | Links | BibTeX | Tags: information theory
@incollection{Lozano2016a,
title = {Fast calculation of entropy with Zhang's estimator},
author = {A. Lozano and B. Casas and C. Bentz and R. Ferrer-i-Cancho},
editor = {J. Macutek R. Knight E. Kelih and A. Wilson},
url = {http://hdl.handle.net/2117/100157},
year = {2016},
date = {2016-01-01},
booktitle = {Issues in Quantitative Linguistics 4. Dedicated to Reinhard Köhler on the occasion of his 65th birthday},
pages = {273-285},
publisher = {RAM-Verlag},
address = {Lüdenscheid},
abstract = {Entropy is a fundamental property of a repertoire. Here, we present an efficient algorithm to estimate the entropy of types with the help of Zhang's estimator. The algorithm takes advantage of the fact that the number of different frequencies in a text is in general much smaller than the number of types. We justify the convenience of the algorithm by means of an analysis of the statistical properties of texts from more than 1000 languages. Our work opens up various possibilities for future research.},
note = {No. 23 of the series ``Studies in Quantitative Linguistic''},
keywords = {information theory},
pubstate = {published},
tppubtype = {incollection}
}
Entropy is a fundamental property of a repertoire. Here, we present an efficient algorithm to estimate the entropy of types with the help of Zhang's estimator. The algorithm takes advantage of the fact that the number of different frequencies in a text is in general much smaller than the number of types. We justify the convenience of the algorithm by means of an analysis of the statistical properties of texts from more than 1000 languages. Our work opens up various possibilities for future research.
Gustison, M. L.; Semple, S.; Ferrer-i-Cancho, R.; Bergman, T.
Gelada vocal sequences follow Menzerath's linguistic law Journal Article
In: Proceedings of the National Academy of Sciences USA, vol. 13, no. 19, pp. E2750–E2758, 2016.
Abstract | Links | BibTeX | Tags: information theory, Menzerath's law
@article{Gustison2016a,
title = {Gelada vocal sequences follow Menzerath's linguistic law},
author = {M. L. Gustison and S. Semple and R. Ferrer-i-Cancho and T. Bergman},
doi = {10.1073/pnas.1522072113},
year = {2016},
date = {2016-01-01},
journal = {Proceedings of the National Academy of Sciences USA},
volume = {13},
number = {19},
pages = {E2750–E2758},
abstract = {Identifying universal principles underpinning diverse natural systems is a key goal of the life sciences. A powerful approach in addressing this goal has been to test whether patterns consistent with linguistic laws are found in nonhuman animals. Menzerath’s law is a linguistic law that states that, the larger the construct, the smaller the size of its constituents. Here, to our knowledge, we present the first evidence that Menzerath’s law holds in the vocal communication of a nonhuman species. We show that, in vocal sequences of wild male geladas (Theropithecus gelada), construct size (sequence size in number of calls) is negatively correlated with constituent size (duration of calls). Call duration does not vary significantly with position in the sequence, but call sequence composition does change with sequence size and most call types are abbreviated in larger sequences. We also find that intercall intervals follow the same relationship with sequence size as do calls. Finally, we provide formal mathematical support for the idea that Menzerath’s law reflects compression—the principle of minimizing the expected length of a code. Our findings suggest that a common principle underpins human and gelada vocal communication, highlighting the value of exploring the applicability of linguistic laws in vocal systems outside the realm of language.},
keywords = {information theory, Menzerath's law},
pubstate = {published},
tppubtype = {article}
}
Identifying universal principles underpinning diverse natural systems is a key goal of the life sciences. A powerful approach in addressing this goal has been to test whether patterns consistent with linguistic laws are found in nonhuman animals. Menzerath’s law is a linguistic law that states that, the larger the construct, the smaller the size of its constituents. Here, to our knowledge, we present the first evidence that Menzerath’s law holds in the vocal communication of a nonhuman species. We show that, in vocal sequences of wild male geladas (Theropithecus gelada), construct size (sequence size in number of calls) is negatively correlated with constituent size (duration of calls). Call duration does not vary significantly with position in the sequence, but call sequence composition does change with sequence size and most call types are abbreviated in larger sequences. We also find that intercall intervals follow the same relationship with sequence size as do calls. Finally, we provide formal mathematical support for the idea that Menzerath’s law reflects compression—the principle of minimizing the expected length of a code. Our findings suggest that a common principle underpins human and gelada vocal communication, highlighting the value of exploring the applicability of linguistic laws in vocal systems outside the realm of language.
Ferrer-i-Cancho, R.
Compression and the origins of Zipf's law for word frequencies Journal Article
In: Complexity, vol. 21, pp. 409-411, 2016.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law for word frequencies
@article{Ferrer2016b,
title = {Compression and the origins of Zipf's law for word frequencies},
author = {R. Ferrer-i-Cancho},
doi = {10.1002/cplx.21820},
year = {2016},
date = {2016-01-01},
journal = {Complexity},
volume = {21},
pages = {409-411},
abstract = {Here we sketch a new derivation of Zipf's law for word frequencies based on optimal coding. The structure of the derivation is reminiscent of Mandelbrot's random typing model but it has multiple advantages over random typing: (1) it starts from realistic cognitive pressures, (2) it does not require fine tuning of parameters, and (3) it sheds light on the origins of other statistical laws of language and thus can lead to a compact theory of linguistic laws. Our findings suggest that the recurrence of Zipf's law in human languages could originate from pressure for easy and fast communication.},
keywords = {information theory, Zipf's law for word frequencies},
pubstate = {published},
tppubtype = {article}
}
Here we sketch a new derivation of Zipf's law for word frequencies based on optimal coding. The structure of the derivation is reminiscent of Mandelbrot's random typing model but it has multiple advantages over random typing: (1) it starts from realistic cognitive pressures, (2) it does not require fine tuning of parameters, and (3) it sheds light on the origins of other statistical laws of language and thus can lead to a compact theory of linguistic laws. Our findings suggest that the recurrence of Zipf's law in human languages could originate from pressure for easy and fast communication.
Ferrer-i-Cancho, R.
The optimality of attaching unlinked labels to unlinked meanings Journal Article
In: Glottometrics, vol. 36, pp. 1-16, 2016.
Abstract | Links | BibTeX | Tags: information theory, vocabulary learning
@article{Ferrer2013g,
title = {The optimality of attaching unlinked labels to unlinked meanings},
author = {R. Ferrer-i-Cancho},
url = {http://hdl.handle.net/2117/102539},
year = {2016},
date = {2016-01-01},
journal = {Glottometrics},
volume = {36},
pages = {1-16},
abstract = {Vocabulary learning by children can be characterized by many biases. When encountering a new word, children as well as adults, are biased towards assuming that it means something totally different from the words that they already know. To the best of our knowledge, the 1st mathematical proof of the optimality of this bias is presented here. First, it is shown that this bias is a particular case of the maximization of mutual information between words and meanings. Second, the optimality is proven within a more general information theoretic framework where mutual information maximization competes with other information theoretic principles. The bias is a prediction from modern information theory. The relationship between information theoretic principles and the principles of contrast and mutual exclusivity is also shown.},
keywords = {information theory, vocabulary learning},
pubstate = {published},
tppubtype = {article}
}
Vocabulary learning by children can be characterized by many biases. When encountering a new word, children as well as adults, are biased towards assuming that it means something totally different from the words that they already know. To the best of our knowledge, the 1st mathematical proof of the optimality of this bias is presented here. First, it is shown that this bias is a particular case of the maximization of mutual information between words and meanings. Second, the optimality is proven within a more general information theoretic framework where mutual information maximization competes with other information theoretic principles. The bias is a prediction from modern information theory. The relationship between information theoretic principles and the principles of contrast and mutual exclusivity is also shown.
2015
Ferrer-i-Cancho, R.; Bentz, C.; Seguin, C.
Compression and the origins of Zipf's law of abbreviation Journal Article
In: 2015.
Links | BibTeX | Tags: information theory, Zipf's law of abbreviation
@article{Ferrer2015a,
title = {Compression and the origins of Zipf's law of abbreviation},
author = {R. Ferrer-i-Cancho and C. Bentz and C. Seguin},
url = {http://arxiv.org/abs/1504.04884},
year = {2015},
date = {2015-01-01},
keywords = {information theory, Zipf's law of abbreviation},
pubstate = {published},
tppubtype = {article}
}
2013
Ferrer-i-Cancho, R.; Hernández-Fernández, A.; Lusseau, D.; Agoramoorthy, G.; Hsu, M. J.; Semple, S.
Compression as a universal principle of animal behavior Journal Article
In: Cognitive Science, vol. 37, no. 8, pp. 1565-1578, 2013.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law of abbreviation
@article{Ferrer2012d,
title = {Compression as a universal principle of animal behavior},
author = {R. Ferrer-i-Cancho and A. Hernández-Fernández and D. Lusseau and G. Agoramoorthy and M. J. Hsu and S. Semple},
doi = {10.1088/1742-5468/2012/06/P06002},
year = {2013},
date = {2013-01-01},
journal = {Cognitive Science},
volume = {37},
number = {8},
pages = {1565-1578},
abstract = {A key aim in biology and psychology is to identify fundamental principles underpinning the behavior of animals, including humans. Analyses of human language and the behavior of a range of non‐human animal species have provided evidence for a common pattern underlying diverse behavioral phenomena: Words follow Zipf's law of brevity (the tendency of more frequently used words to be shorter), and conformity to this general pattern has been seen in the behavior of a number of other animals. It has been argued that the presence of this law is a sign of efficient coding in the information theoretic sense. However, no strong direct connection has been demonstrated between the law and compression, the information theoretic principle of minimizing the expected length of a code. Here, we show that minimizing the expected code length implies that the length of a word cannot increase as its frequency increases. Furthermore, we show that the mean code length or duration is significantly small in human language, and also in the behavior of other species in all cases where agreement with the law of brevity has been found. We argue that compression is a general principle of animal behavior that reflects selection for efficiency of coding.},
keywords = {information theory, Zipf's law of abbreviation},
pubstate = {published},
tppubtype = {article}
}
A key aim in biology and psychology is to identify fundamental principles underpinning the behavior of animals, including humans. Analyses of human language and the behavior of a range of non‐human animal species have provided evidence for a common pattern underlying diverse behavioral phenomena: Words follow Zipf's law of brevity (the tendency of more frequently used words to be shorter), and conformity to this general pattern has been seen in the behavior of a number of other animals. It has been argued that the presence of this law is a sign of efficient coding in the information theoretic sense. However, no strong direct connection has been demonstrated between the law and compression, the information theoretic principle of minimizing the expected length of a code. Here, we show that minimizing the expected code length implies that the length of a word cannot increase as its frequency increases. Furthermore, we show that the mean code length or duration is significantly small in human language, and also in the behavior of other species in all cases where agreement with the law of brevity has been found. We argue that compression is a general principle of animal behavior that reflects selection for efficiency of coding.
Ferrer-i-Cancho, R.; Debowski, L.; Martín, F. Moscoso Prado
Constant conditional entropy and related hypotheses Journal Article
In: Journal of Statistical Mechanics, pp. L07001, 2013.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2013a,
title = {Constant conditional entropy and related hypotheses},
author = {R. Ferrer-i-Cancho and L. Debowski and F. Moscoso Prado Martín},
doi = {10.1088/1742-5468/2013/07/L07001},
year = {2013},
date = {2013-01-01},
journal = {Journal of Statistical Mechanics},
pages = {L07001},
abstract = {Constant entropy rate (conditional entropies must remain constant as the sequence length increases) and uniform information density (conditional probabilities must remain constant as the sequence length increases) are two information theoretic principles that are argued to underlie a wide range of linguistic phenomena. Here we revise the predictions of these principles in the light of Hilberg's law on the scaling of conditional entropy in language and related laws. We show that constant entropy rate (CER) and two interpretations for uniform information density (UID), full UID and strong UID, are inconsistent with these laws. Strong UID implies CER but the reverse is not true. Full UID, a particular case of UID, leads to costly uncorrelated sequences that are totally unrealistic. We conclude that CER and its particular cases are incomplete hypotheses about the scaling of conditional entropies.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Constant entropy rate (conditional entropies must remain constant as the sequence length increases) and uniform information density (conditional probabilities must remain constant as the sequence length increases) are two information theoretic principles that are argued to underlie a wide range of linguistic phenomena. Here we revise the predictions of these principles in the light of Hilberg's law on the scaling of conditional entropy in language and related laws. We show that constant entropy rate (CER) and two interpretations for uniform information density (UID), full UID and strong UID, are inconsistent with these laws. Strong UID implies CER but the reverse is not true. Full UID, a particular case of UID, leads to costly uncorrelated sequences that are totally unrealistic. We conclude that CER and its particular cases are incomplete hypotheses about the scaling of conditional entropies.
2012
Ferrer-i-Cancho, R.; McCowan, B.
The span of dependencies in dolphin whistle sequences Journal Article
In: Journal of Statistical Mechanics, pp. P06002, 2012.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2012c,
title = {The span of dependencies in dolphin whistle sequences},
author = {R. Ferrer-i-Cancho and B. McCowan},
doi = {10.1088/1742-5468/2012/06/P06002},
year = {2012},
date = {2012-01-01},
journal = {Journal of Statistical Mechanics},
pages = {P06002},
abstract = {Long-range correlations are found in symbolic sequences from human language, music and DNA. Determining the span of correlations in dolphin whistle sequences is crucial for shedding light on their communicative complexity. Dolphin whistles share various statistical properties with human words, i.e. Zipf's law for word frequencies (namely that the probability of the ith most frequent word of a text is about i-a) and a parallel of the tendency of more frequent words to have more meanings. The finding of Zipf's law for word frequencies in dolphin whistles has been the topic of an intense debate on its implications. One of the major arguments against the relevance of Zipf's law in dolphin whistles is that it is not possible to distinguish the outcome of a die-rolling experiment from that of a linguistic or communicative source producing Zipf's law for word frequencies. Here we show that statistically significant whistle–whistle correlations extend back to the second previous whistle in the sequence, using a global randomization test, and to the fourth previous whistle, using a local randomization test. None of these correlations are expected by a die-rolling experiment and other simple explanations of Zipf's law for word frequencies, such as Simon's model, that produce sequences of unpredictable elements.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Long-range correlations are found in symbolic sequences from human language, music and DNA. Determining the span of correlations in dolphin whistle sequences is crucial for shedding light on their communicative complexity. Dolphin whistles share various statistical properties with human words, i.e. Zipf's law for word frequencies (namely that the probability of the ith most frequent word of a text is about i-a) and a parallel of the tendency of more frequent words to have more meanings. The finding of Zipf's law for word frequencies in dolphin whistles has been the topic of an intense debate on its implications. One of the major arguments against the relevance of Zipf's law in dolphin whistles is that it is not possible to distinguish the outcome of a die-rolling experiment from that of a linguistic or communicative source producing Zipf's law for word frequencies. Here we show that statistically significant whistle–whistle correlations extend back to the second previous whistle in the sequence, using a global randomization test, and to the fourth previous whistle, using a local randomization test. None of these correlations are expected by a die-rolling experiment and other simple explanations of Zipf's law for word frequencies, such as Simon's model, that produce sequences of unpredictable elements.
2011
Ferrer-i-Cancho, R.; Martín, F. Moscoso Prado
Information content versus word length in random typing Journal Article
In: Journal of Statistical Mechanics, pp. L12002, 2011.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2011c,
title = {Information content versus word length in random typing},
author = {R. Ferrer-i-Cancho and F. Moscoso Prado Martín},
doi = {10.1088/1742-5468/2011/12/L12002},
year = {2011},
date = {2011-01-01},
journal = {Journal of Statistical Mechanics},
pages = {L12002},
abstract = {Recently, it has been claimed that a linear relationship between a measure of information content and word length is expected from word length optimization and it has been shown that this linearity is supported by a strong correlation between information content and word length in many languages (Piantadosi et al 2011 Proc. Nat. Acad. Sci. 108 3825). Here, we study in detail some connections between this measure and standard information theory. The relationship between the measure and word length is studied for the popular random typing process where a text is constructed by pressing keys at random from a keyboard containing letters and a space behaving as a word delimiter. Although this random process does not optimize word lengths according to information content, it exhibits a linear relationship between information content and word length. The exact slope and intercept are presented for three major variants of the random typing process. A strong correlation between information content and word length can simply arise from the units making a word (e.g., letters) and not necessarily from the interplay between a word and its context as proposed by Piantadosi and co-workers. In itself, the linear relation does not entail the results of any optimization process.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Recently, it has been claimed that a linear relationship between a measure of information content and word length is expected from word length optimization and it has been shown that this linearity is supported by a strong correlation between information content and word length in many languages (Piantadosi et al 2011 Proc. Nat. Acad. Sci. 108 3825). Here, we study in detail some connections between this measure and standard information theory. The relationship between the measure and word length is studied for the popular random typing process where a text is constructed by pressing keys at random from a keyboard containing letters and a space behaving as a word delimiter. Although this random process does not optimize word lengths according to information content, it exhibits a linear relationship between information content and word length. The exact slope and intercept are presented for three major variants of the random typing process. A strong correlation between information content and word length can simply arise from the units making a word (e.g., letters) and not necessarily from the interplay between a word and its context as proposed by Piantadosi and co-workers. In itself, the linear relation does not entail the results of any optimization process.
2008
Ferrer-i-Cancho, R.
Information theory Book Section
In: Hogan, P. Colm (Ed.): The Cambridge encyclopedia of the language sciences, Cambridge University Press, 2008.
BibTeX | Tags: information theory
@incollection{Ferrer2008b,
title = {Information theory},
author = {R. Ferrer-i-Cancho},
editor = {P. Colm Hogan},
year = {2008},
date = {2008-01-01},
booktitle = {The Cambridge encyclopedia of the language sciences},
publisher = {Cambridge University Press},
keywords = {information theory},
pubstate = {published},
tppubtype = {incollection}
}
2007
Ferrer-i-Cancho, R.
On the universality of Zipf's law for word frequencies Book Section
In: Grzybek, P.; Köhler, R. (Ed.): Exact methods in the study of language and text. To honor Gabriel Altmann, pp. 131-140, Gruyter, Berlin, 2007.
Links | BibTeX | Tags: information theory, Zipf's law for word frequencies
@incollection{Ferrer2006a,
title = {On the universality of Zipf's law for word frequencies},
author = {R. Ferrer-i-Cancho},
editor = {P. Grzybek and R. Köhler},
doi = {10.1515/9783110894219.131},
year = {2007},
date = {2007-01-01},
booktitle = {Exact methods in the study of language and text. To honor Gabriel Altmann},
pages = {131-140},
publisher = {Gruyter},
address = {Berlin},
keywords = {information theory, Zipf's law for word frequencies},
pubstate = {published},
tppubtype = {incollection}
}
Ferrer-i-Cancho, R.; Díaz-Guilera, A.
The global minima of the communicative energy of natural communication systems Journal Article
In: Journal of Statistical Mechanics, pp. P06009, 2007.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2007a,
title = {The global minima of the communicative energy of natural communication systems},
author = {R. Ferrer-i-Cancho and A. Díaz-Guilera},
doi = {10.1088/1742-5468/2007/06/P06009},
year = {2007},
date = {2007-01-01},
journal = {Journal of Statistical Mechanics},
pages = {P06009},
abstract = {Until recently, models of communication have explicitly or implicitly assumed that the goal of a communication system is just maximizing the information transfer between signals and 'meanings'. Recently, it has been argued that a natural communication system not only has to maximize this quantity but also has to minimize the entropy of signals, which is a measure of the cognitive cost of using a word. The interplay between these two factors, i.e. maximization of the information transfer and minimization of the entropy, has been addressed previously using a Monte Carlo minimization procedure at zero temperature. Here we derive analytically the globally optimal communication systems that result from the interaction between these factors. We discuss the implications of our results for previous studies within this framework. In particular we prove that the emergence of Zipf's law using a Monte Carlo technique at zero temperature in previous studies indicates that the system had not reached the global optimum.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Until recently, models of communication have explicitly or implicitly assumed that the goal of a communication system is just maximizing the information transfer between signals and 'meanings'. Recently, it has been argued that a natural communication system not only has to maximize this quantity but also has to minimize the entropy of signals, which is a measure of the cognitive cost of using a word. The interplay between these two factors, i.e. maximization of the information transfer and minimization of the entropy, has been addressed previously using a Monte Carlo minimization procedure at zero temperature. Here we derive analytically the globally optimal communication systems that result from the interaction between these factors. We discuss the implications of our results for previous studies within this framework. In particular we prove that the emergence of Zipf's law using a Monte Carlo technique at zero temperature in previous studies indicates that the system had not reached the global optimum.
2006
Ferrer-i-Cancho, R.
When language breaks into pieces. A conflict between communication through isolated signals and language Journal Article
In: Biosystems, vol. 84, pp. 242-253, 2006.
Abstract | Links | BibTeX | Tags: information theory, network science
@article{Ferrer2005e,
title = {When language breaks into pieces. A conflict between communication through isolated signals and language},
author = {R. Ferrer-i-Cancho},
doi = {10.1016/j.biosystems.2005.12.001},
year = {2006},
date = {2006-01-01},
journal = {Biosystems},
volume = {84},
pages = {242-253},
abstract = {Here, we study a communication model where signals associate to stimuli. The model assumes that signals follow Zipf’s law and the exponent of the law depends on a balance between maximizing the information transfer and saving the cost of signal use. We study the effect of tuning that balance on the structure of signal–stimulus associations. The model starts from two recent results. First, the exponent grows as the weight of information transfer increases. Second, a rudimentary form of language is obtained when the network of signal–stimulus associations is almost connected. Here, we show the existence of a sudden destruction of language once a critical balance is crossed. The model shows that maximizing the information transfer through isolated signals and language are in conflict. The model proposes a strong reason for not finding large exponents in complex communication systems: language is in danger. Besides, the findings suggest that human words may need to be ambiguous to keep language alive. Interestingly, the model predicts that large exponents should be associated to decreased synaptic density. It is not surprising that the largest exponents correspond to schizophrenic patients since, according to the spirit of Feinberg’s hypothesis, i.e. decreased synaptic density may lead to schizophrenia. Our findings suggest that the exponent of Zipf’s law is intimately related to language and that it could be used to detect anomalous structure and organization of the brain.},
keywords = {information theory, network science},
pubstate = {published},
tppubtype = {article}
}
Here, we study a communication model where signals associate to stimuli. The model assumes that signals follow Zipf’s law and the exponent of the law depends on a balance between maximizing the information transfer and saving the cost of signal use. We study the effect of tuning that balance on the structure of signal–stimulus associations. The model starts from two recent results. First, the exponent grows as the weight of information transfer increases. Second, a rudimentary form of language is obtained when the network of signal–stimulus associations is almost connected. Here, we show the existence of a sudden destruction of language once a critical balance is crossed. The model shows that maximizing the information transfer through isolated signals and language are in conflict. The model proposes a strong reason for not finding large exponents in complex communication systems: language is in danger. Besides, the findings suggest that human words may need to be ambiguous to keep language alive. Interestingly, the model predicts that large exponents should be associated to decreased synaptic density. It is not surprising that the largest exponents correspond to schizophrenic patients since, according to the spirit of Feinberg’s hypothesis, i.e. decreased synaptic density may lead to schizophrenia. Our findings suggest that the exponent of Zipf’s law is intimately related to language and that it could be used to detect anomalous structure and organization of the brain.
Ferrer-i-Cancho, R.; Lusseau, D.
Long-term correlations in the surface behavior of dolphins Journal Article
In: Europhysics Letters, vol. 74, no. 6, pp. 1095-1101, 2006.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2005h,
title = {Long-term correlations in the surface behavior of dolphins},
author = {R. Ferrer-i-Cancho and D. Lusseau},
doi = {10.1209/epl/i2005-10596-9},
year = {2006},
date = {2006-01-01},
journal = {Europhysics Letters},
volume = {74},
number = {6},
pages = {1095-1101},
abstract = {Here we study the sequences of surface behavioral patterns of dolphins (Tursiops sp.) and find long-term correlations. We show that the long-term correlations are not of a trivial nature, i.e. they cannot be explained by the repetition of the same surface behavior many times in a row. Our findings suggest that dolphins have a long collective memory extending back at least to the 7-th past behavior. As far as we know, this is the first evidence of long-term correlations in the behavior of a non-human species.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Here we study the sequences of surface behavioral patterns of dolphins (Tursiops sp.) and find long-term correlations. We show that the long-term correlations are not of a trivial nature, i.e. they cannot be explained by the repetition of the same surface behavior many times in a row. Our findings suggest that dolphins have a long collective memory extending back at least to the 7-th past behavior. As far as we know, this is the first evidence of long-term correlations in the behavior of a non-human species.
2005
Ferrer-i-Cancho, R.
Decoding least effort and scaling in signal frequency distributions Journal Article
In: Physica A, vol. 345, pp. 275-284, 2005.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law for word frequencies
@article{Ferrer2003c,
title = {Decoding least effort and scaling in signal frequency distributions},
author = {R. Ferrer-i-Cancho},
doi = {10.1016/j.physa.2004.06.158},
year = {2005},
date = {2005-01-01},
journal = {Physica A},
volume = {345},
pages = {275-284},
abstract = {Here, assuming a general communication model where objects map to signals, a power function for the distribution of signal frequencies is derived. The model relies on the satisfaction of the receiver (hearer) communicative needs when the entropy of the number of objects per signal is maximized. Evidence of power distributions in a linguistic context (some of them with exponents clearly different from the typical $beta approx 2$ of Zipf's law) is reviewed and expanded. We support the view that Zipf's law reflects some sort of optimization but following a novel realistic approach where signals (e.g. words) are used according to the objects (e.g. meanings) they are linked to. Our results strongly suggest that many systems in nature use non-trivial strategies for easing the interpretation of a signal. Interestingly, constraining just the number of interpretations of signals does not lead to scaling.},
keywords = {information theory, Zipf's law for word frequencies},
pubstate = {published},
tppubtype = {article}
}
Here, assuming a general communication model where objects map to signals, a power function for the distribution of signal frequencies is derived. The model relies on the satisfaction of the receiver (hearer) communicative needs when the entropy of the number of objects per signal is maximized. Evidence of power distributions in a linguistic context (some of them with exponents clearly different from the typical $beta approx 2$ of Zipf's law) is reviewed and expanded. We support the view that Zipf's law reflects some sort of optimization but following a novel realistic approach where signals (e.g. words) are used according to the objects (e.g. meanings) they are linked to. Our results strongly suggest that many systems in nature use non-trivial strategies for easing the interpretation of a signal. Interestingly, constraining just the number of interpretations of signals does not lead to scaling.
Ferrer-i-Cancho, R.
The variation of Zipf's law in human language Journal Article
In: European Physical Journal B, vol. 44, pp. 249-257, 2005.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2004a,
title = {The variation of Zipf's law in human language},
author = {R. Ferrer-i-Cancho},
doi = {10.1140/epjb/e2005-00121-8},
year = {2005},
date = {2005-01-01},
journal = {European Physical Journal B},
volume = {44},
pages = {249-257},
abstract = {Words in humans follow the so-called Zipf’s law. More precisely, the word frequency spectrum follows a power function, whose typical exponent is β≈2, but significant variations are found. We hypothesize that the full range of variation reflects our ability to balance the goal of communication, i.e. maximizing the information transfer and the cost of communication, imposed by the limitations of the human brain. We show that the higher the importance of satisfying the goal of communication, the higher the exponent. Here, assuming that words are used according to their meaning we explain why variation in β should be limited to a particular domain. From the one hand, we explain a non-trivial lower bound at about β=1.6 for communication systems neglecting the goal of the communication. From the other hand, we find a sudden divergence of β if a certain critical balance is crossed. At the same time a sharp transition to maximum information transfer and unfortunately, maximum communication cost, is found. Consistently with the upper bound of real exponents, the maximum finite value predicted is about β=2.4. It is convenient for human language not to cross the transition and remain in a domain where maximum information transfer is high but at a reasonable cost. Therefore, only a particular range of exponents should be found in human speakers. The exponent β contains information about the balance between cost and communicative efficiency.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Words in humans follow the so-called Zipf’s law. More precisely, the word frequency spectrum follows a power function, whose typical exponent is β≈2, but significant variations are found. We hypothesize that the full range of variation reflects our ability to balance the goal of communication, i.e. maximizing the information transfer and the cost of communication, imposed by the limitations of the human brain. We show that the higher the importance of satisfying the goal of communication, the higher the exponent. Here, assuming that words are used according to their meaning we explain why variation in β should be limited to a particular domain. From the one hand, we explain a non-trivial lower bound at about β=1.6 for communication systems neglecting the goal of the communication. From the other hand, we find a sudden divergence of β if a certain critical balance is crossed. At the same time a sharp transition to maximum information transfer and unfortunately, maximum communication cost, is found. Consistently with the upper bound of real exponents, the maximum finite value predicted is about β=2.4. It is convenient for human language not to cross the transition and remain in a domain where maximum information transfer is high but at a reasonable cost. Therefore, only a particular range of exponents should be found in human speakers. The exponent β contains information about the balance between cost and communicative efficiency.
Ferrer-i-Cancho, R.
Zipf's law from a communicative phase transition Journal Article
In: European Physical Journal B, vol. 47, pp. 449-457, 2005.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2004e,
title = {Zipf's law from a communicative phase transition},
author = {R. Ferrer-i-Cancho},
doi = {10.1140/epjb/e2005-00340-y},
year = {2005},
date = {2005-01-01},
journal = {European Physical Journal B},
volume = {47},
pages = {449-457},
abstract = {Here we present a new model for Zipf's law in human word frequencies. The model defines the goal and the cost of communication using information theory. The model shows a continuous phase transition from a no communication to a perfect communication phase. Scaling consistent with Zipf's law is found in the boundary between phases. The exponents are consistent with minimizing the entropy of words. The model differs from a previous model [Ferrer i Cancho, Solé, Proc. Natl. Acad. Sci. USA 100, 788–791 (2003)] in two aspects. First, it assumes that the probability of experiencing a certain stimulus is controlled by the internal structure of the communication system rather than by the probability of experiencing it in the `outside' world, which makes it specially suitable for the speech of schizophrenics. Second, the exponent α predicted for the frequency versus rank distribution is in a range where α>1, which may explain that of some schizophrenics and some children, with α=1.5-1.6. Among the many models for Zipf's law, none explains Zipf's law for that particular range of exponents. In particular, two simplistic models fail to explain that particular range of exponents: intermittent silence and Simon's model. We support that Zipf's law in a communication system may maximize the information transfer under constraints.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Here we present a new model for Zipf's law in human word frequencies. The model defines the goal and the cost of communication using information theory. The model shows a continuous phase transition from a no communication to a perfect communication phase. Scaling consistent with Zipf's law is found in the boundary between phases. The exponents are consistent with minimizing the entropy of words. The model differs from a previous model [Ferrer i Cancho, Solé, Proc. Natl. Acad. Sci. USA 100, 788–791 (2003)] in two aspects. First, it assumes that the probability of experiencing a certain stimulus is controlled by the internal structure of the communication system rather than by the probability of experiencing it in the `outside' world, which makes it specially suitable for the speech of schizophrenics. Second, the exponent α predicted for the frequency versus rank distribution is in a range where α>1, which may explain that of some schizophrenics and some children, with α=1.5-1.6. Among the many models for Zipf's law, none explains Zipf's law for that particular range of exponents. In particular, two simplistic models fail to explain that particular range of exponents: intermittent silence and Simon's model. We support that Zipf's law in a communication system may maximize the information transfer under constraints.
Ferrer-i-Cancho, R.
Hidden communication aspects inside the exponent of Zipf's law Journal Article
In: Glottometrics, vol. 11, pp. 96-117, 2005.
BibTeX | Tags: information theory
@article{Ferrer2005d,
title = {Hidden communication aspects inside the exponent of Zipf's law},
author = {R. Ferrer-i-Cancho},
year = {2005},
date = {2005-01-01},
journal = {Glottometrics},
volume = {11},
pages = {96-117},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
2003
Ferrer-i-Cancho, R.; Solé, R. V.
Least effort and the origins of scaling in human language Journal Article
In: Proceedings of the National Academy of Sciences USA, vol. 100, pp. 788-791, 2003.
Abstract | Links | BibTeX | Tags: information theory, Zipf's law for word frequencies
@article{Ferrer2002a,
title = {Least effort and the origins of scaling in human language},
author = {R. Ferrer-i-Cancho and R. V. Solé},
doi = {10.1073/pnas.0335980100},
year = {2003},
date = {2003-01-01},
journal = {Proceedings of the National Academy of Sciences USA},
volume = {100},
pages = {788-791},
abstract = {The emergence of a complex language is one of the fundamental events of human evolution, and several remarkable features suggest the presence of fundamental principles of organization. These principles seem to be common to all languages. The best known is the so-called Zipf's law, which states that the frequency of a word decays as a (universal) power law of its rank. The possible origins of this law have been controversial, and its meaningfulness is still an open question. In this article, the early hypothesis of Zipf of a principle of least effort for explaining the law is shown to be sound. Simultaneous minimization in the effort of both hearer and speaker is formalized with a simple optimization process operating on a binary matrix of signal–object associations. Zipf's law is found in the transition between referentially useless systems and indexical reference systems. Our finding strongly suggests that Zipf's law is a hallmark of symbolic reference and not a meaningless feature. The implications for the evolution of language are discussed. We explain how language evolution can take advantage of a communicative phase transition.},
keywords = {information theory, Zipf's law for word frequencies},
pubstate = {published},
tppubtype = {article}
}
The emergence of a complex language is one of the fundamental events of human evolution, and several remarkable features suggest the presence of fundamental principles of organization. These principles seem to be common to all languages. The best known is the so-called Zipf's law, which states that the frequency of a word decays as a (universal) power law of its rank. The possible origins of this law have been controversial, and its meaningfulness is still an open question. In this article, the early hypothesis of Zipf of a principle of least effort for explaining the law is shown to be sound. Simultaneous minimization in the effort of both hearer and speaker is formalized with a simple optimization process operating on a binary matrix of signal–object associations. Zipf's law is found in the transition between referentially useless systems and indexical reference systems. Our finding strongly suggests that Zipf's law is a hallmark of symbolic reference and not a meaningless feature. The implications for the evolution of language are discussed. We explain how language evolution can take advantage of a communicative phase transition.
Ferrer-i-Cancho, R.
Language: universals, principles and origins PhD Thesis
Universitat Politècnica de Catalunya, 2003.
Abstract | BibTeX | Tags: information theory, network science
@phdthesis{Ferrer2003b,
title = {Language: universals, principles and origins},
author = {R. Ferrer-i-Cancho},
year = {2003},
date = {2003-01-01},
address = {Barcelona},
school = {Universitat Politècnica de Catalunya},
abstract = {Here, old and new linguistic universals, i.e. properties obeyed by all languages on Earth are investigated. Basic principles of language predicting linguistic universals are also investigated. More precisely, two principles of reference, i.e. coding least effort and decoding least effort, a reformulation of G. K. Zipf's speaker and hearer least effort principles. Such referential principles predict Zipf's law, a universal of word frequencies, at the maximum tension between coding and decoding needs. Although trivial processes have been proposed for explaining Zipf's law in non-linguistic contexts, Zipf's law meaningfulness for human language is supported here. Minimizing the Euclidean distance between syntactically related words in sentences is a principle predicting projectivity, a universal stating that arcs between syntactically linked words in sentences generally do not cross. Besides, such a physical distance minimization successfully predicts (a) an exponential distribution for the distribution of the distance between syntactically related words and (b) subject-verb-object (SVO) order superiority in the actual use of world languages. Previously unreported non-trivial features of real syntactic dependency networks are presented here, i.e. scale-free degree distributions, small-world phenomenon, disassortative mixing and hierarchical organization. Instead of a universal grammar, a single universality class is proposed for world languages.
Syntax and symbolic reference are unified under a single topological property, ie. connectedness in the network of signal-object associations of a communication system. Assuming Zipf's law, not only connectedness follows, but the above properties of real syntactic networks. Therefore, (a) referential principles are the principles of syntax and symbolic reference, (b) syntax is a by product of simple communication principles and (c) the above properties of syntactic dependency networks must be universal if Zipf's law is universal, which is the case. The transition to language is shown to be of the kind of a continuous phase transition in physics. Thereafter, the transition to human language could not have been gradual. The reduced network morphospace resulting from a combination of a network distance minimization principle and link density minimization principle is presented as an alternative hypothesis and a promising prospect for linguistic networks subject to fast communication pressures.
The present thesis is unique among theories about the origins of language, in the sense that (a) it explains how words or signals naturally glue in order to form complex messages, (b) it validates its predictions with real data, (c) unifies syntax and symbolic reference and (d) uses ingredients already present in the animal communication systems, in a way no other approximations do. The framework presented is radical shift in the research of linguistic universals and its origins through the physics of critical phenomena. The principles presented here are not principles of human language, but principles of complex communication. Therefore, the such principles suggest new prospects for other information transmission systems in nature.},
keywords = {information theory, network science},
pubstate = {published},
tppubtype = {phdthesis}
}
Here, old and new linguistic universals, i.e. properties obeyed by all languages on Earth are investigated. Basic principles of language predicting linguistic universals are also investigated. More precisely, two principles of reference, i.e. coding least effort and decoding least effort, a reformulation of G. K. Zipf's speaker and hearer least effort principles. Such referential principles predict Zipf's law, a universal of word frequencies, at the maximum tension between coding and decoding needs. Although trivial processes have been proposed for explaining Zipf's law in non-linguistic contexts, Zipf's law meaningfulness for human language is supported here. Minimizing the Euclidean distance between syntactically related words in sentences is a principle predicting projectivity, a universal stating that arcs between syntactically linked words in sentences generally do not cross. Besides, such a physical distance minimization successfully predicts (a) an exponential distribution for the distribution of the distance between syntactically related words and (b) subject-verb-object (SVO) order superiority in the actual use of world languages. Previously unreported non-trivial features of real syntactic dependency networks are presented here, i.e. scale-free degree distributions, small-world phenomenon, disassortative mixing and hierarchical organization. Instead of a universal grammar, a single universality class is proposed for world languages.
Syntax and symbolic reference are unified under a single topological property, ie. connectedness in the network of signal-object associations of a communication system. Assuming Zipf's law, not only connectedness follows, but the above properties of real syntactic networks. Therefore, (a) referential principles are the principles of syntax and symbolic reference, (b) syntax is a by product of simple communication principles and (c) the above properties of syntactic dependency networks must be universal if Zipf's law is universal, which is the case. The transition to language is shown to be of the kind of a continuous phase transition in physics. Thereafter, the transition to human language could not have been gradual. The reduced network morphospace resulting from a combination of a network distance minimization principle and link density minimization principle is presented as an alternative hypothesis and a promising prospect for linguistic networks subject to fast communication pressures.
The present thesis is unique among theories about the origins of language, in the sense that (a) it explains how words or signals naturally glue in order to form complex messages, (b) it validates its predictions with real data, (c) unifies syntax and symbolic reference and (d) uses ingredients already present in the animal communication systems, in a way no other approximations do. The framework presented is radical shift in the research of linguistic universals and its origins through the physics of critical phenomena. The principles presented here are not principles of human language, but principles of complex communication. Therefore, the such principles suggest new prospects for other information transmission systems in nature.
Syntax and symbolic reference are unified under a single topological property, ie. connectedness in the network of signal-object associations of a communication system. Assuming Zipf's law, not only connectedness follows, but the above properties of real syntactic networks. Therefore, (a) referential principles are the principles of syntax and symbolic reference, (b) syntax is a by product of simple communication principles and (c) the above properties of syntactic dependency networks must be universal if Zipf's law is universal, which is the case. The transition to language is shown to be of the kind of a continuous phase transition in physics. Thereafter, the transition to human language could not have been gradual. The reduced network morphospace resulting from a combination of a network distance minimization principle and link density minimization principle is presented as an alternative hypothesis and a promising prospect for linguistic networks subject to fast communication pressures.
The present thesis is unique among theories about the origins of language, in the sense that (a) it explains how words or signals naturally glue in order to form complex messages, (b) it validates its predictions with real data, (c) unifies syntax and symbolic reference and (d) uses ingredients already present in the animal communication systems, in a way no other approximations do. The framework presented is radical shift in the research of linguistic universals and its origins through the physics of critical phenomena. The principles presented here are not principles of human language, but principles of complex communication. Therefore, the such principles suggest new prospects for other information transmission systems in nature.
2002
Ferrer-i-Cancho, R.; Reina, F.
Quantifying the semantic contribution of particles Journal Article
In: Journal of Quantitative Linguistics, vol. 9, pp. 35-47, 2002.
Abstract | Links | BibTeX | Tags: information theory
@article{Ferrer2002f,
title = {Quantifying the semantic contribution of particles},
author = {R. Ferrer-i-Cancho and F. Reina},
doi = {10.1076/jqul.9.1.35.8483},
year = {2002},
date = {2002-01-01},
journal = {Journal of Quantitative Linguistics},
volume = {9},
pages = {35-47},
abstract = {Certain word types of natural languages - conjunctions, articles, prepositions and some verbs - have a very low or very grammatically marked semantic contribution. They are usually named functional categories or relational items. Recently, the possibility of considering prepositions as simple parametrical variations of semantic features instead of categorial features or as the irrelevance of such categorial features has been pointed out. The discussion about such particles has been and still is widespread and controversial. Nonetheless, there is no quantitative evidence of such semantic weakness and no satisfactory evidence against the coexistence of categorial requirements and the fragility of the semantic aspects. This study aims to quantify the semantic contribution of particles and presents some corpora-based results for English that suggest that such weakness and its relational uncertainty come from the categorial irrelevance mentioned before.},
keywords = {information theory},
pubstate = {published},
tppubtype = {article}
}
Certain word types of natural languages - conjunctions, articles, prepositions and some verbs - have a very low or very grammatically marked semantic contribution. They are usually named functional categories or relational items. Recently, the possibility of considering prepositions as simple parametrical variations of semantic features instead of categorial features or as the irrelevance of such categorial features has been pointed out. The discussion about such particles has been and still is widespread and controversial. Nonetheless, there is no quantitative evidence of such semantic weakness and no satisfactory evidence against the coexistence of categorial requirements and the fragility of the semantic aspects. This study aims to quantify the semantic contribution of particles and presents some corpora-based results for English that suggest that such weakness and its relational uncertainty come from the categorial irrelevance mentioned before.
In case the fancy publication browser above fails, you can also try.