Home | Introduction | What's Included | Products | More Info | FAQ's

≥ Marktplaats - Cookiewall gekleurde timberland laarzen

Marktplaats.nl gebruikt functionele, analytische en tracking cookies (en daarmee vergelijkbare technieken) om jouw ervaring op onze we qouwxdla. 赤いティンバーランドbsite te verbeteren en om je van relevante advertenties te voorzien.

gekleurde timberland laarzen

женские лесные сапоги дешево
timberland winter boots
waar te koop timberland laarzen

Gender Recognition on Dutch Tweets

Maat: px Weergave met pagina beginnen:

Download "Gender Recognition on Dutch Tweets"
  • Willem Moens
  • 2 jaren geleden
  • Aantal bezoeken:


1 Computational Linguistics in the Netherlands Journal 4 (2014) Submitted 06/2014; Published 12/2014 Gender Recognition on Dutch Tweets Hans van Halteren Nander Speerstra Radboud University Nijmegen, CLS, Linguistics Abstract In this paper, we investigate gender recognition on Dutch Twitter material, using a corpus consisting of the full Tweet production (as far as present in the TwiNL data set) of 600 users (known to be human individuals) over 2011 and We experimented with several authorship profiling techniques and various recognition features, using Tweet text only, in order to determine how well they could distinguish between male and female authors of Tweets. We achieved the best results, 95.5% correct assignment in a 5-fold cross-validation on our corpus, with Support Vector Regression on all token unigrams. Two other machine learning systems, Linguistic Profiling and TiMBL, come close to this result, at least when the input is first preprocessed with PCA. 1. Introduction In the Netherlands, we have a rather unique resource in the form of the TwiNL data set: a daily updated collection that probably contains at least 30% of the Dutch public tweet production since 2011 (Tjong Kim Sang and van den Bosch 2013). However, as any collection that is harvested automatically, its usability is reduced by a lack of reliable metadata. In this case, the Twitter profiles of the authors are available, but these consist of freeform text rather than fixed information fields. And, obviously, it is unknown to which degree the information that is present is true. The resource would become even more useful if we could deduce complete and correct metadata from the various available information sources, such as the provided metadata, user relations, profile photos, and the text of the tweets. In this paper, we start modestly, by attempting to derive just the gender of the authors 1 automatically, purely on the basis of the content of their tweets, using author profiling techniques. For our experiment, we selected 600 authors for whom we were able to determine with a high degree of certainty a) that they were human individuals and b) what gender they were. We then experimented with several author profiling techniques, namely Support Vector Regression (as provided by LIBSVM; (Chang and Lin 2011)), Linguistic Profiling (LP; (van Halteren 2004)), and TiMBL (Daelemans et al. 2004), with and without preprocessing the input vectors with Principal Component Analysis (PCA; (Pearson 1901); (Hotelling 1933)). We also varied the recognition features provided to the techniques, using both character and token n-grams. For all techniques and features, we ran the same 5-fold cross-validation experiments in order to determine how well they could be used to distinguish between male and female authors of tweets. In the following sections, we first present some previous work on gender recognition (Section 2). Then we describe our experimental data and the evaluation method (Section 3), after which we proceed to describe the various author profiling strategies that we investigated (Section 4). Then follow the results (Section 5), and Section 6 concludes the paper. 1. For whom we already know that they are an individual person rather than, say, a husband and wife couple or a board of editors for an official Twitterfeed. c 2014 van Halteren and Speerstra.

2 2. Gender Recognition Gender recognition is a subtask in the general field of authorship recognition and profiling, which has reached maturity in the last decades(for an overview, see e.g. (Juola 2008) and (Koppel et al. 2009)). Currently the field is getting an impulse for further development now that vast data sets of user generated data is becoming available. Narayanan et al. (2012) show that authorship recognition is also possible (to some degree) if the number of candidate authors is as high as 100,000 (as compared to the usually less than ten in traditional studies). Even so, there are circumstances where outright recognition is not an option, but where one must be content with profiling, i.e. the identification of author traits like gender, age and geographical background. In this paper we restrict ourselves to gender recognition, and it is also this aspect we will discuss further in this section. A group which is very active in studying gender recognition (among other traits) on the basis of text is that around Moshe Koppel. In (Koppel et al. 2002) they report gender recognition on formal written texts taken from the British National Corpus (and also give a good overview of previous work), reaching about 80% correct attributions using function words and parts of speech. Later, in 2004, the group collected a Blog Authorship Corpus (BAC; (Schler et al. 2006)), containing about 700,000 posts to blogger.com (in total about 140 million words) by almost 20,000 bloggers. For each blogger, metadata is present, including the blogger s self-provided gender, age, industry and astrological sign. This corpus has been used extensively since. The creators themselves used it for various classification tasks, including gender recognition (Koppel et al. 2009). They report an overall accuracy of 76.1%. Slightly more information seems to be coming from content (75.1% accuracy) than from style (72.0% accuracy). However, even style appears to mirror content. We see the women focusing on personal matters, leading to important content words like love and boyfriend, and important style words like I and other personal pronouns. The men, on the other hand, seem to be more interested in computers, leading to important content words like software and game, and correspondingly more determiners and prepositions. One gets the impression that gender recognition is more sociological than linguistic, showing what women and men were blogging about back in A later study (Goswami et al. 2009) managed to increase the gender recognition quality to 89.2%, using sentence length, 35 non-dictionary words, and 52 slang words. The authors do not report the set of slang words, but the non-dictionary words appear to be more related to style than to content, showing that purely linguistic behaviour can contribute information for gender recognition as well. Gender recognition has also already been applied to Tweets. Rao et al. (2010) examined various traits of authors from India tweeting in English, combining character N-grams and sociolinguistic features like manner of laughing, honorifics, and smiley use. With lexical N-grams, they reached an accuracy of 67.7%, which the combination with the sociolinguistic features increased to 72.33%. Burger et al. (2011) attempted to recognize gender in tweets from a whole set of languages, using word and character N-grams as features for machine learning with Support Vector Machines (SVM), Naive Bayes and Balanced Winnow2. Their highest score when using just text features was 75.5%, testing on all the tweets by each author (with a train set of 3.3 million tweets and a test set of about 418,000 tweets). 2 Fink et al. (2012) used SVMlight to classify gender on Nigerian twitter accounts, with tweets in English, with a minimum of 50 tweets. Their features were hash tags, token unigrams and psychometric measurements provided by the Linguistic Inquiry of Word Count software (LIWC; (Pennebaker et al. 2007)). Although LIWC appears a very interesting addition, it hardly adds anything to the classification. With only token unigrams, the recognition accuracy was 80.5%, while using all features together increased this only slightly to 80.6%. Bamman et al. (2014) examined about 9 million tweets by 14,000 Twitter users tweeting in American English. They used lexical features, and present a very good breakdown of various word types. When using all user tweets, they reached an accuracy of 88.0%. An interesting observation is that there is a clear class of misclassified users who have a majority of opposite gender users in their social network. 2. When adding more information sources, such as profile fields, they reach an accuracy of 92.0%. 172

3 For Tweets in Dutch, we first look at the official user interface for the TwiNL data set, Among other things, it shows gender and age statistics for the users producing the tweets found for user specified searches. These statistics are derived from the users profile information by way of some heuristics. For gender, the system checks the profile for about 150 common male and 150 common female first names, as well as for gender related words, such as father, mother, wife and husband. If no cue is found in a user s profile, no gender is assigned. The general quality of the assignment is unknown, but in the (for this purpose) rather unrepresentative sample of users we considered for our own gender assignment corpus (see below), we find that about 44% of the users are assigned a gender, which is correct in about 87% of the cases. Another system that predicts the gender for Dutch Twitter users is TweetGenie (http://www.tweetgenie.nl), that one can provide with a Twitter user name, after which the gender and age are estimated, based on the user s last 200 tweets. The age component of the system is described in (Nguyen et al. 2013). The authors apply logistic and linear regression on counts of token unigrams occurring at least 10 times in their corpus. The paper does not describe the gender component, but the first author has informed us that the accuracy of the gender recognition on the basis of 200 tweets is about 87% (Nguyen, personal communication). 3 In later experiments, Nguyen et al. (2014) did a crowdsourcing experiment, in which they asked human participants to guess the gender and age on the basis of 20 to 40 tweets. When using a majority vote to represent the crowd s opinion, the crowd s perception of the gender on the basis of the tweets coincided with the actual gender in about 84% of the cases. The conclusion is not so much, however, that humans are also not perfect at guessing age on the basis of language use, but rather that there is a distinction between the biological and the social identity of authors, and language use is more likely to represent the social one (cf. also (Bamman et al. 2014)). Although we agree with Nguyen et al. on this, we will still take the biological gender as the gold standard in this paper, as our eventual goal is creating metadata for the TwiNL collection. 3. Experimental Data and Evaluation In this section, we first describe the corpus that we used in our experiments (Section 3.1). Then we outline how we evaluated the various strategies (Section 3.2). 3.1 Corpus Used in the Experiments We selected our experimental material from the TwiNL data set (Tjong Kim Sang and van den Bosch 2013), which was collected by searching for tweets with any of a number of probably Dutch words, after which a character n-gram language filter was applied. The collection is estimated to contain 30-40% of all public Dutch tweets. From this material, we considered all tweets with a date stamp in 2011 and In all, there were about 23 million users present. Of these, we only considered the ones who produced 2 to 10 tweets on average per day over 2011 and The minimum ensured a sufficient amount of text ( tweets) for classification; the maximum served to avoid very high volume users, who might be professional (multi-user/edited) feeds or even twitterbots. This restriction brought the number of users down to about 270,000. We then progressed to the selection of individual users. We aimed for 600 users. We selected 500 of these so that they get a gender assignment in TwiQS, for comparison, but we also wanted to include unmarked users in case these would be different in nature. All users, obviously, should be individuals, and for each the gender should be clear. From the about 120,000 users who are assigned a gender by TwiQS, we took a random selection in such a manner that the volume distribution (i.e. from 2 to 10 tweets per day average) is equally spread throughout the range and approximately equal for men and women. We checked gender manually for all selected users, mostly on the basis 3. As in our own experiment, this measurement is based on Twitter accounts where the user is known to be a human individual. 173

4 of the profile texts and profile photo s, and only included those for which we were convinced of the gender. 4 Later even more detailed rechecks, after a few extremely unlikely classification results, served to clean up the (hopefully) last gender assignment errors. 5 The final corpus is not completely balanced for gender, but consists of the production of 320 women and 280 men. However, as research shows a higher number of female users in all as well (Heil and Piskorski 2009), we do not view this as a problem. From each user s tweets, we removed all retweets, as these did not contain original text by the author. Then, as several of our features were based on tokens, we tokenized all text samples, using our own specialized tokenizer for tweets. Apart from normal tokens like words, numbers and dates, it is also able to recognize a wide variety of emoticons. The tokenizer is able to identify hashtags and Twitter user names to the extent that these conform to the conventions used in Twitter, i.e. the hash (#) resp. at sign are followed by a series of letters, digits and underscores. URLs and addresses are not completely covered. The tokenizer counts on clear markers for these, e.g. http, www or one of a number of domain names for URLs. Assuming that any sequence including periods is likely to be a URL provesunwise, given that spacing between normal wordsis often irregular. And actually checking the existence of a proposed URL was computationally infeasible for the amount of text we intended to process. Finally, as the use of capitalization and diacritics is quite haphazard in the tweets, the tokenizer strips all words of diacritics and transforms them to lower case. 3.2 Evaluation We divided our corpus in five parts, each containing (approximately) the same number of male and female authors. 6 We used this division in all experiments, each time using four parts as training material and one as test material. For those techniques where hyperparameters need to be selected, we used a leave-one-out strategy on the test material. For each test author, we determined the optimal hyperparameter settings with regard to the classification of all other authors in the same part of the corpus, in effect using these as development material. In this way, we derived a classification score for each author without the system having any direct or indirect access to the actual gender of the author. We then measured for which percentage of the authors in the corpus this score was in agreement with the actual gender. These percentages are presented below in Section Profiling Strategies In this section, we describe the strategies that we investigated for the gender recognition task. As we approached the task from a machine learning viewpoint, we needed to select text features to be provided as input to the machine learning systems, as well as machine learning systems which are to use this input for classification. We first describe the features we used (Section 4.1). Then we explain how we used the three selected machine learning systems to classify the authors (Section 4.2). 4.1 Machine Learning Features We restricted ourselves to lexical features for our experiments. The use of syntax or even higher level features is (for now) impossible as the language use on Twitter deviates too much from standard Dutch, and we have no tools to provide reliable analyses. However, even with purely lexical features, 4. On the examined users, the gender assignment of TwiQS proved about 87% correct. 5. Several errors could be traced back to the fact that the account had moved on to another user since We could have used different dividing strategies, but chose balanced folds in order to give a equal chance to all machine learning techniques, also those that have trouble with unbalanced data. If, in any application, unbalanced collections are expected, the effects of biases, and corrections for them, will have to be investigated. 174

5 there are still various options from which to choose. Most of them rely on the tokenization described above. We will illustrate the options we explored with the Hahaha...ik geloof dat meneer Bee heeft ingezet op een plan vóór het slapen ;-))ng hahaha... I believe that mister B has opted for a plan before sleeping <smiley> ng which after preprocessing becomes hahaha... ik geloof dat meneer bee heeft ingezet op een plan voor het slapen <smiley> ng The first option for machine learning features is a traditional one. Top 100 Function Words The most frequent function words(see(kestemont 2014) for an overview). We used the 100 most frequent, as measured on our tweet collection, of which the example tweet contains the words ik, dat, heeft, op, een, voor, and het. Then, we used a set of feature types based on token n-grams, with which we already had previous experience (Van Bael and van Halteren 2007). For all feature types, we used only those features which were observed with at least 5 authors in our whole collection (for skip bigrams 10 authors). Unigrams Single tokens, similar to the top function words, but then using all tokens instead of a subset. About 47K features. In the example tweet, we find e.g. ik, ingezet, and <smiley>. Bigrams Two adjacent tokens. About 265K features. In the example tweet, e.g. heeft ingezet, slapen <smiley>, and where the double underscore represents the start of the tweet. Trigrams Three adjacent tokens. About 355K features. In the example tweet, e.g. op een plan and hahaha. Skip bigrams Two tokens in the tweet, but not adjacent, without any restrictions on the gap size. About 580K features. In the example tweet, e.g. dat heeft and hahaha <smiley>. Finally, we included feature types based on character n-grams(following(kjell et al. 1994)). We used the n-grams with n from 1 to 5, again only when the n-gram was observed with at least 5 authors. However, we used two types of character n-grams. The first set is derived from the tokenizer output, and can be viewed as a kind of normalized character n-grams. Normalized 1-gram About 350 features. In the example tweet, e.g. i Normalized 2-gram About 4K features. In the example tweet, e.g. ik and twice ng. Normalized 3-gram About 36K features. In the example tweet, e.g. gez and n v, where the underscore represents a space. Normalized 4-gram About 160K features. In the example tweet, e.g. slap and op e. Normalized 5-gram About 420K features. In the example tweet, e.g. ingez and now with a double underscore for the beginning of the tweet and a single underscore for a space. The second set of character n-grams is derived from the original tweets. This type of character n-gram has the clear advantage of not needing any preprocessing in the form of tokenization. Original 1-gram About 420 features. In the example tweet, e.g. e and ;. 175

6 Original 2-gram About 8K features. In the example tweet, e.g. Be Original 3-gram About 77K features. In the example tweet, e.g. ing and ))n. Original 4-gram About 260K features. In the example tweet, e.g. plan and ;-)). Original 5-gram About 580K features. In the example tweet, e.g. r Bee and a...i. 4.2 Machine Learning Techniques Having determined the features we would be working with, we next needed to select a machine learning system. Again, we decided to explore more than one option, but here we preferred more focus and restricted ourselves to three systems. Our primary choice for classification was the use of Support Vector Machines, viz. LIBSVM (Chang and Lin, 2001). We chose Support Vector Regression (ν-svr to be exact) with an RBF kernel, as it had shown the best results in several research projects (e.g. (van Halteren 2008)). With these main choices, we performed a grid search for well-performing hyperparameters, with the following investigated values: the cost factor C is set to respectively 1/32, 1, 32, 1024, and 32768, γ to 1/4, 1/2, 1, 2 and 4 times LIBSVM s default of one divided by the number of features, and ν to 0.1, 0.3, 0.5 and 0.7. The second classification system was Linguistic Profiling (LP; (van Halteren 2004)), which was specifically designed for authorship recognition and profiling. Roughly speaking, it classifies on the basis of noticeable over- and underuse of specific features. Before being used in comparisons, all feature counts were normalized to counts per 1000 words, and then transformed to Z-scores with regard to the average and standard deviation within each feature. LP has four hyperparameter settings, three of which weight the relative importance of each feature/dimension in the feature vector when comparing a text s feature vector to the profile vector (in this case the average of the feature vectors for all the training texts for a given gender), and one determining the threshold for feature Z- scores to be taken into account. Here the grid search investigated: the hyperparameter emphasizing the difference between text feature and profile feature to polynomial exponents set to 0.1, 0.4, 0.7,..., 2.7 and 3; the hyperparameters for emphasizing text feature size to 0 or 1; the hyperparameter for emphasizing profile feature size to -1, 0, 1, and 2; and the threshold hyperparameter also to 0 or 1. Finally, we added TiMBL (Daelemans et al. 2004), a k-nearest neighbour classification system, which is used extensively in-house for various machine learning tasks, but which we had so far not used for authorship tasks. As the input features are numerical, we used IB1 with k equal to 5 so that we can derive a confidence value. The only hyperparameters we varied in the grid search are the metric (Numerical and Cosine distance) and the weighting (no weighting, information gain, gain ratio, chi-square, shared variance, and standard deviation). However, the high dimensionality of our vectors presented us with a problem. For such high numbers of features, it is known that k-nn learning is unlikely to yield useful results (Beyer et al. 1999). This meant that, if we still wanted to use k-nn, we would have to reduce the dimensionality of our feature vectors. We chose to use Principal Component Analysis (PCA; (Pearson 1901), (Hotelling 1933)). 7 And, now we had the principal component vectors, we decided also to provide them to SVR and LP. For each system, we provided the first N principal components for various N. In effect, this N is a further hyperparameter, which we varied from 1 to the total number of components (usually 600, as there are 600 authors), using a stepsize of 1 from 1 to 10, and then slowly increasing the stepsize to a maximum of 20 when over 300. Rather than using fixed hyperparameters, we let the control shell choose them automatically in a grid search procedure, based on development data. When running the underlying systems 7. To be exact, we used the function prcomp in R (R Development Core Team 2008), with the instruction scale = T RUE to force normalization of the vectors before the principal components were determined. As scaling is not possible when there are columns with constant values, such columns were removed first. 176

7 themselves, we used various hyperparameter settings, as listed above. For each setting and author, the systems report both a selected class and a floating point score, which can be used as a confidence score. 8 For each individual author, the control shell examined the scores for all other authors in the samefold. 9 Itthencalculatedaclassseparationvalue, namelythedifferencebetweenthemeanscores for each of the two classes (male and female), divided by the sum of the two standard deviations. 10 The optimal hyperparameter settings are assumed to be those where the two classes are separated most, i.e. where the class separation value is highest. In order to improve the robustness of the hyperparameter selection, the best three settings were chosen and used for classifying the current author in question. A final detail that we exploited is that SVR and LP are asymmetric in the modeling of the classes. For LP, this is by design. A model, called profile, is constructed for each individual class, and the system determines for each author to which degree they are similar to the class profile. For SVR, one would expect symmetry, as both classes are modeled simultaneously, and differ merely in the sign of the numeric class identifier. However, we do observe different behaviour when reversing the signs. For this reason, we did all classification with SVR and LP twice, once building a male model and once a female model. For both models the control shell calculated a final score, starting with the three outputs for the best hyperparameter settings. It normalized these by expressing them as the number of non-model class standard deviations over the threshold, which was set at the class separation value. The control shell then weighted each score by multiplying it by the class separation value on the development data for the settings in question, and derived the final score by averaging. It then chose the class for which the final score is highest. In this way, we also get two confidence values, viz. the model score for the chosen class (how male/female the author writes) and the difference between the two scores (how much more female/male the author writes than male/female). 5. Results In this section, we will present the overall results of the gender recognition. We start with the accuracy of the various features and systems (Section 5.1). Then we will focus on the effect of preprocessing the input vectors with PCA (Section 5.2). After this, we examine the classification of individual authors (Section 5.3 and the distinguishing power of features (Section 5.4). 5.1 Overall Quality Table 1 shows the accuracy of the recognition, using the desccribed features and systems. For the systems, both SVR and LP are used with the original case vectors as well as with PCA preprocessing, where TiMBL, for reasons mentioned above, is used only with preprocessed vectors. For the measurements with PCA, the number of principal components provided to the classification system is learned from the development data. Below, in Section 5.2, we will examine what the systems are capable of at fixed numbers of principal components. Starting with the systems, we see that SVR (using original vectors) consistently outperforms the other two. For only one feature type, character trigrams, LP with PCA manages to reach a higher accuracy than SVR, but the difference is not statistically significant. LP and TiMBL are closely matched, although LP appears to be slightly better when combined with PCA, but the next section will shed new light on this comparison. From the measurements here, we can conclude that LP profits from PCA preprocessing, but SVR is better off with the original vectors. 8. For SVR and LP, these are rather varied, but TiMBL s confidence value consists of the proportion of selected class cases among the nearest neighbours, which with k at 5 is practically always 0.6, 0.8, or This gives the best chances that the selected optimal hyperparameters generalize to the author in question. 10. The class separation value is a variant of Cohen s d (Cohen 1988). Where Cohen assumes the two distributions have the same standard deviation, we use the sum of the two, practically always different, standard deviations. 177

8 Table 1: Accuracy Percentages for various Feature Types and Techniques. For each feature type, the best percentage is bolded, and all percentages are italicized that are not statistically significantly different at the 5% level. Feature type Techniques Support Vector Regression Linguistic Profiling TiMBL original with PCA original with PCA with PCA Top 100 Function Words Token Unigram Token Bigram Token Trigram Token Skip Bigram Char 1-gram-n Char 2-gram-n Char 3-gram-n Char 4-gram-n Char 5-gram-n Char 1-gram-o Char 2-gram-o Char 3-gram-o Char 4-gram-o Char 5-gram-o As for features types, the token unigrams are clearly the best choice. In fact, for all the tokens n-grams, it would seem that the further one goes away from the unigrams, the worse the accuracy gets. An explanation for this might be that recognition is mostly on the basis of the content of the tweet, and unigrams represent the content most clearly. Possibly, the other n-grams are just mirroring this quality of the unigrams, with the effectiveness of the mirror depending on how well unigrams are represented in the n-grams. Below (Section 5.4), we will have a closer look at this hypothesis. For the character n-grams, our first observation is that the normalized versions are always better than the original versions. This means that the content of the n-grams is more important than their form. This is in accordance with the hypothesis just suggested for the token n-grams, as normalization too brings the character n-grams closer to token unigrams. The best performing character n-grams (normalized 5-grams), will be most closely linked to the token unigrams, with some token bigrams thrown in, as well as a smidgen of the use of morphological processes. However, we cannot conclude that what is wiped away by the normalization, use of diacritics, capitals and spacing, holds no information for the gender recognition. To test that, we would have to experiment with a new feature types, modeling exactly the difference between the normalized and the original form. 5.2 Effects of PCA In the measurements above, the number of principal components provided to the classification systems was learned on the basis of the development sets. This number was treated as just another hyperparameter to be selected. As a result, the systems accuracy was partly dependent on the quality of the hyperparameter selection mechanism. In this section, we want to investigate how strong this dependency may have been. 178

9 Attribution Accuracy SVM LP TiMBL Log2(Number of Principal Components) Figure 1: Recognition accuracy as a function of the number of principal components provided to the systems, using token unigrams. The dotted line represents the accuracy of SVR without PCA preprocessing. Figures 1, 2, and 3 show accuracy measurements for the token unigrams, token bigrams, and normalized character 5-grams, for all three systems at various numbers of principal components. The dotted line is at the accuracy of SVR without PCA. For the unigrams, SVR reaches its peak (94.7%) around principal components, with a second peak around TiMBL closely follows SVR, but only reaches its best score (94.2%) at the latter peak (80-90). Interestingly, it is SVR that degrades at higher numbers of principal components, while TiMBL, said to need fewer dimensions, manages to hold on to the recognition quality. LP peaks much earlier (93.8%) at only 8-10 principal components. However, it does not manage to achieve good results with the principal components that were best for the other two systems. Furthermore, LP appears to suffer some kind of mathematical breakdown for higher numbers of components. If we look at these measurements, it would seem we should prefer TiMBL over LP, which is in contradiction to what we see in Table 1. Although LP performs worse than it could on fixed numbers of principal components, its more detailed confidence score allows a better hyperparameter selection, on average selecting around 9 principal components, where TiMBL chooses a wide range of numbers, and generally far lower than is optimal. We expect that the performance with TiMBL can be improved greatly with the development of a better hyperparameter selection mechanism. For the bigrams (Figure 2), we see much the same picture, although there are differences in the details. SVR now already reaches its peak (94.3%) at 10 principal components, and stays at almost the same quality until around 200. TiMBL peaks a bit later at 200 with 94.7%, even slightly higher than SVR without PCA. And LP just mirrors its behaviour with unigrams. For the normalized character 5-grams, SVR is clearly better than TiMBL, with peaks (94.2%) from 40 to 100. LP keeps its peak at 10, but now even lower than for the token n-grams (92.8%). All in all, we can conclude that SVR without PCA is still the best choice. However, all systems are in principle able to reach the same quality (i.e. not significantly lower) with the optimal number of principal components. Even with an automatically selected number, LP already profits clearly 179

10 Attribution Accuracy SVM LP TiMBL Log2(Number of Principal Components) Figure 2: Recognition accuracy as a function of the number of principal components provided to the systems, using token bigrams. The dotted line represents the accuracy of SVR without PCA preprocessing. from PCA, but (for this task) lags behind. And TiMBL is currently underperforming, but might be a challenger to SVR when provided with a better hyperparameter selection mechanism. 5.3 Analysis of Author Classifications In this section, we will examine some aspects of author classifications. We will focus on the token n-grams and the normalized character 5-grams. As for systems, we will involve all five systems in the discussion. However, our starting point will always be SVR with token unigrams, this being the best performing combination. We will only look at the final scores for each combination, and forgo the extra detail of any underlying separate male and female model scores (which we have for SVR and LP; see above). As can be seen in Figure 4, the two scores for SVR match almost completely anyway (Pearson Correlation ). 11 The major exception to the symmetry is author 543, lying clearly in the male area, but quite a bit above the dotted line (at around -2,4 in Figure 4). When we look at his tweets, we see a kind of financial blog, which is an exception in the population we have in our corpus. The exception also leads to more varied classification by the different systems, yielding a wide range of scores. SVR tends to place him clearly in the male area with all the feature types, with unigrams at the extreme with a score of SVR with PCA on the other hand, is less convinced, and even classifies him as female for unigrams (1.136) and skipgrams (3.946). LP and TiMBL also show scores all over the range. Figure 4 shows that the male population contains some more extreme exponents than the female population. The most obvious male is author 430, with a resounding Looking at his texts, we indeed see a prototypical (young) male Twitter user: the addressed topics mainly consist of soccer, gaming, school, and music (all of which we will see again below, when examining the most gender 11. This is rather different for LP, but the focus is on SVR here. 12. From this point on in the discussion, we will present female confidence as positive numbers and male as negative. 180

11 Attribution Accuracy SVM LP TiMBL Log2(Number of Principal Components) Figure 3: Recognition accuracy as a function of the number of principal components provided to the systems, using normalized character 5-grams. The dotted line represents the accuracy of SVR without PCA preprocessing. specific unigrams). All systems have no trouble recognizing him as a male, with the lowest scores (around 1) for the top 100 function words. If we look at the rest of the top males (Table 2), we may see more varied topics, but the wide recognizability stays. Unigrams are mostly closely mirrored by the character 5-grams, as could already be suspected from the content of these two feature types. For the other feature types, we see some variation, but most scores are found near the top of the lists. Table 2: Top ranking males in SVR on token unigrams, with ranks and scores for SVR with various feature types. Feature type Unigram 1: : : : : Bigram 1: : : : : Trigram 1: : : : : Skipgram 1: : : : : Char 5-gram 1: : : : : Top 100 Function 4: : : : : On the female side, everything is less extreme. The best recognizable female, author 264, is not as focused as her male counterpart. There is much more variation in the topics, but most of it is clearly girl talk (of the type described in Section 5.4), again putting the best recognition at a prototypical young Twitter user. In scores, too, we see far more variation. Even the character 5-grams have ranks up to 40 for this top-5. Another interesting group of authors is formed by the misclassified ones. Taking again SVR on unigrams as our starting point, this group contains 11 males and 16 females. We show the 5 most 181

12 Score from Female Profile Score from Male Profile Figure 4: Confidence scores for gender assignment with regard to the female and male profiles built by SVR on the basis of token unigrams. The dashed line represents the separation threshold, i.e. a higher score for one gender than for the other. The dotted line represents exactly opposite scores for the two genders. 182

13 Table 3: Top rankingfemales insvr ontokenunigrams, with ranksand scoresforsvr with various feature types. Feature type Unigram 1: : : : : Bigram 6: : : : : Trigram 43: : : : : Skipgram 10: : : : : Char 5-gram 2: : : : : Top 100 Function 9: : : : : Table 4: Most strongly misclassified males in SVR on token unigrams, with scores for SVR with various feature types. Feature type Unigram Bigram Trigram Skipgram Char 5-gram Top 100 Function strongly misclassified ones of each gender in Tables 4 and 5. With one exception (author 355 is recognized as male when using trigrams), all feature types agree on the misclassification. This may support ourhypothesis that allfeature types aredoingmore orlessthe same. But it might alsomean that the gender just influences all feature types to a similar degree. In addition, the recognition is of course also influenced by our particular selection of authors, as we will see shortly. Apart from the general agreement on the final decision, the feature types vary widely in the scores assigned, but this also allows for both conclusions. The male which is attributed the most female score is author 352. On (re)examination, we see a clearly male first name and also profile photo. However, his Twitter network contains mostly female friends. This apparently colours not only the discussion topics, which might be expected, but also the general language use. 13 Another interesting case is author 389. The unigrams do not judge him to write in an extremely female way, but all other feature types do. When looking at his tweets, we 13. This has also been remarked by Bamman et al. (2014). Table 5: Most strongly misclassified females in SVR on token unigrams, with scores for SVR with various feature types. Feature type Unigram Bigram Trigram Skipgram Char 5-gram Top 100 Function

14 get the impression that Dutch is not his native language, which is supported by his name. There is an extreme number of misspellings (even for Twitter), which may possibly confuse the systems models. The most extreme misclassification is reserved for a female, author 103. This turns out to be Judith Sargentini, a member of the European Parliament, who tweets under the 14 Although clearly female, she is judged as rather strongly male (-3.285) when using unigrams and character 5-grams, and male by all system-feature combinations 15 except three. LP with PCA on skipgrams assigns her a female score of and SVR with PCA (just as with author 543) arrives at a clearly female attribution with character 5-grams (4.554) and unigrams (5.149). In this case, it would seem that the systems are thrown off by the political texts. If we search for the word parlement ( parliament ) in our corpus, which is used 40 times by Sargentini, we find two more female authors (each using it once), as compared to 21 male authors (with up to 9 uses). Apparently, in our sample, politics is a male thing. 16 It is intriguing that both here and with the male financial blogger, the erroneous misclassification with unigrams is reversed when using PCA on the unigrams. We did a quick spot check with author 113, a girl who plays soccer and is therefore also misclassified often; here, the PCA version agrees with and misclassified even stronger than the original unigrams ( versus ). In later research, when we will try to identify the various user types on Twitter, we will certainly have another look at this phenomenon. 5.4 Features In the analysis so far, we have wondered several times what kind of features are responsible for the rather accurate classification. Are they mostly targeting the content of the tweets, i.e. related to the activities of the authors in real life, or the style, i.e. the way they use the basic building blocks of the Dutch language? In this section, we will attempt to get closer to the answer to this question. Again, we take the token unigrams as a starting point. However, looking at SVR is not an option here. Because of the way in which SVR does its classification, hyperplane separation in a transformed version of the vector space, it is impossible to determine which features do the most work. Instead, we will just look at the distribution of the various features over the female and male texts. Figure 5 shows all token unigrams. The ones used more by women are plotted in green, those used more by men in red. The position in the plot represents the relative number of men and women who used the token at least once somewhere in their tweets. However, for classification, it is more important how often the token is used by each gender. We represent this quality by the class separation value that we described in Section 4.2, and show it in the form of font size, i.e. the more distinguishing tokens are bigger. As the separation value and the percentages are generally correlated, the bigger tokens are found further away from the diagonal, while the area close to the diagonal contains mostly unimportant (and therefore unreadable) tokens. On the female side, we see a representation of the world of the prototypical young female Twitter user. Itisaveryemotionalplace,withomg ( OhMyGod! ) inacentralposition, butalsocontaining giggling (hihi) and lots of emotionally loaded adjectives, such as lief and lieve ( sweet ), schattig ( cute ), leuk and leuke ( nice ). And also some more negative emotions, such as haat ( hate ) and pijn ( pain ). Next we see personal care, with nagels ( nails ), nagellak ( nail polish ), makeup ( makeup ), mascara ( mascara ), and krullen ( curls ). Clearly, shopping is also important, as is watching soaps on television (gtst). The age is reconfirmed by the endearingly high presence of mama and papa. As for style, the only real factor is echt ( really ). The word haar may be the pronoun her, but just as well the noun hair, and in both cases it is actually more related to the 14. Identity disclosed with permission. 15. And by TweetGenie as well. 16. An alternative hypothesis was that Sargentini does not write her own tweets, but assigns this task to a male press spokesperson. However, we received confirmation that she writes almost all her tweets herself (Sargentini, personal communication). 184

15 Percentage of males using token Percentage of females using token ik, je! en? niet de een is <smiley> het dat op maar in met me van nog naar dan rt heb wel die voor ook zo nu weer... ben echt : haha wat DQUO te als er ga bij moet al jij kan aan ja zijn lekker slapen was we x = morgen ze <url> geen om doen gaan!! ; uit mijn wil wakker ( mij goed toch dus gaat ff eten $ even uur school meer weet of jou vandaag kijken over 3 gewoon = mn thuis hoor xd nee ) wie dit _ heeft k net 2 hahaha leuk veel t kom mensen hebben o denk waar 1 hij eens he alleen hoe komt mee man zin zit hier altijd oke maken had heel nou & tot door < daar bed dag bent doe leren helemaal niks jaa kut?? alles weg vanavond word jullie iets doet werken terug klaar tegen deze laat mag vind waarom huis zou beetje iemand bijna straks want hem komen na ofzo haar hou nieuwe zie zeg m=n iedereen half anders tijd 5 snel / moe nooit best omdat hele mooi zal twitter beter worden keer steeds hahah hebt gezellig allemaal 4 vroeg ging moeten wij zien wordt amp xx staat week kunnen jaar zeker vrij wachten 10 gwn kijk af pas erg vakantie eerst zeggen tv werk halen krijg gehad nie s toen zegt 8 <url> eindelijk omg mis zitten pff toe tweet lang gedaan 6 oh moeder ons andere daarna d zelf kapot zo=n laatste buiten schatje dood 7 okee gelukkig film eerste bb haat hahahaha slaap der hihi dacht dingen sorry eigenlijk super lief blij leven volgens weten hoop leuke gek alweer maakt hard =t ken voel snap weekend bedje wanneer enzo eh ie douchen msn staan geweest gister foto heerlijk 9 zei binnen verder fiets achter kk gelijk dr zonder mama wilt bus moest koud ping misschien avond lopen mooie soms laten niemand volgende lieve gemaakt // egt tweets hoelaat 12 praten auto muziek beneden paar engels dagen zag schat ma fietsen jammer gezien stad <symb> liggen +twexit ziek nodig zometeen kamer beginnen elke les alle hun vanmiddag n onder maak vrijdag ok pijn bezig uurtje heen wiskunde richting mobiel trainen m dagje vast oma raar jaaa das fijn meisje elkaar willen zaterdag == welke klas normaal tog sommige vraag beste boven lijkt samen joh ouders maandag kon hoofd honger stage ligt a idee denken zon trein warm wauw wtf hahahah da grappig stuk vragen blijven voetbal pauze vader kwart nummer neem vergeten ziet ava proberen langs valt ajax geld wedstrijd 11 bank lol lig geslapen zn xxx geef vet slecht serieus via praat gegeten zetten geven plezier zat minuten trouwens goede telefoon spelen pakken duits vinden onze miss broertje zusje boos zij lachen idd wou frans blijft drinken opeens zoveel krijgen douche papa retweet internet rustig begint waren genoeg nederlands druk dom 1e 0 sta wist pfff 00 blijf houden verjaardag klopt l later shit huiswerk kopen graag eigen ah kwam morge vrienden jongen dm zondag min ding + jarig natuurlijk hoef vol qq precies vanaf naast effe begin niets p 20 bedoel verveel hw rond dikke wrm naam denkt hey vriend weken zich tas zoo saai jongens voetballen regen terwijl grote bellen ogen vaak foto=s gesprek z=n u nederland laptop 15 broer kanker wow moeilijk heet fucking nemen twee dankje training tussen ooh succes vond krijgt euro zus kwijt open wekker hopen chill liedje meteen bio pa gvd lezen donderdag gevoel opa uurtjes + fuck bang eerder kun zoals chillen kleine top hond toets geloof bedankt werd vrouw onderweg deed goeie goedemorgen wacht zet feest hoeveel opstaan stoer hmm love wereld lange moment ergens w jouw hadden nieuw my minder ooit ophalen geeft sws mezelf omkleden rijden hahahahaha vanaaf i doei luisteren whatsapp hopelijk maarja eet hoezo ander 30 alsof tering zoeken jonge <starred> boek woensdag ineens neee gym loopt vallen ofso bek the kind oude bel dinsdag klein hea schoenen aardig % gisteren rest broek liefde geweldig gefeliciteerd lukt > et geworden ipod jy dank gekregen gtst nacht grapje baby hee horen zwemmen weinig groot stuur facking ruzie hoi vertellen ochtend fifa nl dicht iphone q ak vriendin last koffie nice avondje ow zielig lach echte gegaan amsterdam nergens leeg daarom winnen volg whaha zomaar klaarmaken helaas zelfs stil kinderen stond geleden gewonnen 2e xo tweeten huilen houd slaapt enige deur dn eruit ene zoek eng brengen benieuwd kkr broodje kop alvast dik +gn aankleden voorbij tl droog vroeger gekocht lelijk meisjes zullen ey ver mogen boeken gehaald opruimen klote pizza gemist rug familie so em cool hand loop lees filmpje oud meid sturen hoofdpijn pc online tis vindt kapper yes jezus vnv sowieso zingen regent voelt app haal geleerd jezelf dalijk gevonden helpen hyves tuurlijk ahaha chille missen zwaar hoeft feestje hehe jaloers werkt huh kermis dansen c waarschijnlijk you 100 sinds e inderdaad geschiedenis ontbijten volgend los kent eventjes mac aah woorden 16 kijkt yeah hart sms vertel pak genieten kans liever tijdje water by ach erbij wilde val volgen gedoucht naja zelfde duurt per mop weetje 13 vorige teveel lui site probleem fout kort juist irritant rare hy bitch vies afgelopen gingen dokter xd. +moe o. liefje soort vergeet roken slim uitslapen =s totaal zomer kant rust tijdens 50 gratis bah thee inpakken kreeg station gekke +tienerfeiten mond feyenoord strand jas eentje rooster psv mam fijne lkkr liefste vooral haren hoorde computer team bn zooo bier kunt woord g jawel reclame mja wa lekkere hoort kleren den overal eerlijk kei jah verwacht middag zyn graden trots 14 home j maand stoppen vannacht langer viel luister uitval tijden 33 voordat gezet jaja gezicht helft wss =n drie dt winkel ut dromen tuin zonnetje spijt thanks stukje reageer afspreken dankjewel gehoord blackberry stem hahahahah damn antwoord tante facebook maat nagels hb hahaa economie gekomen lag gooi makkelijk vraagt kerst w=app sneeuw gekeken klinkt gezegd aaah straat ongeveer misselijk gooien =m probeer licht gebeld pingen jaaaa hahha verhaal slaan mr +linkdedownie ha kwartier spullen gevallen whahaha zodat schatjes btw speelt xxxx live nogsteeds gs eey neus godverdomme nieuws kheb pw examen slaat tja dorp erop ziekenhuis buikpijn mocht mannen begonnen trek mail f tosti gestuurd opnieuw utrecht spreken droom koop 18 lukken aant stiekem verkeerd fv on liedjes slape wifi behalve vrolijk zouden hetzelfde prima maanden ohh reageren uhm to kga raam lul nr stom zeer tandarts hahahha 40 koken boys rood meneer kennen snapt verloren warme hebbe moeite shirt omw namelijk okeee ookal knap geval god +dtv verliefd plan +durftevragen erin gebeurt programma spreek binnenkort boy downloaden geluk scooter baas ontbijt vorig nix kijke heeel wapp noem domme plek hallo persoon leuks gebruiken schrijven never myn lieverd gaa spel link woow vrouwen 17 gebeuren welk meen club focking nichtje aandacht fack ~ 25 kat oog brood keertje vieze neemt hoer enz 3e ouwe leer noemen politie respect veranderen oo up r eigelijk homo vervelen ni ijsje regenen nat schiet erger ghehe schattig cijfer blijkbaar no groningen vn jong ofz mijne oooh wint leef oook mams geniet keihard wind nii eraan zwembad gelezen s. ahh slechte einde dadelijk bios ahah gast plaats extra reageert twente mens tenminste groep grond rot speciaal ehh goeiemorgen houdt meenemen woon ps3 iris natuurkunde lastig ipad problemen pffff 2012 zoiets rotterdam gegeven zorgen +slapen geniaal ohja gebeurd rij durf vervelend reden ieder se gewerkt h benen optijd vertrouwen korte bedoelde onzin sinterklaas ef punten woont vuurwerk lucht oom sexy betalen tafel oppassen zwart km ijs chips stel land punt radio zoon taart weeer pitten bad wonen +haha manier camping hahahahahaha boeit tussenuur spanje grap relatie tegenwoordig godver geloven osso handig donker +not meestal account toetsen handen oja koppijn scheikunde opzich hmmm vriendje vieren buik b maaar sterkte allebei tweede meiden elk zak geslaagd +tweetoff gamen stress volgensmij mogelijk veranderd 1ste pap wc stomme vanochtend potje z eco vaker reactie +lekker ortho eind begrijp rennen dames voelen wijf tanden ipv seizoen leuker ahahaha stop bestaat schijnt verslag it zoooo rii toevallig bal witte 45 buurt trap hangen [ men leg kamp biertje patat degene tien deel ] duidelijk mt welkom volgt held neefje kerel sukkel oplader waard bericht leraar boo heerlijke poging zwarte vd sloom hoog verkeerde zulke buren 19 ll v tim ei halve sporten zolang belt fc verpest vriendinnetje zeuren sport dezelfde ervan ontbijtje regelen check help vent 22 hardlopen nja xdd presentatie gebruikt opmaken duitsland kaartje gesprekken +tweetbreak ai batterij gebruik vanuit twitteren helpt nek betekent logeren gespeeld back mist gay 24 laatst knie wees geluid xoxo bye nick saaie oeps speel neeee 21 +tvoh moesten jan rode lieffie bakken altyd liep ri gebracht allang allen zegge nouja bestellen aha moppie kont d. tof blauwe oranje volgers weekje boodschappen netjes harde gezegt vliegen haag kiezen liegen laag gm zichzelf hihihi es alsnog gesproken waarheid duur pop kids samenvatting spannend d=r hah follow merk opschieten youtube draaien keek thuiss eff indd nask zooi neef tekenen uren arm tege heletijd dronken meeste 4e schaatsen hoelang slaaplekker +geenzin diegene tram kast vier cijfers kaartjes langzaam bak skype verslaafd daan voeten aanwezig vanacht kill stadje stappen jo getweet flikker shoppen direct aa kip da=s sla noemt max +kenjedat grootste balen vreemd ge brengt +goodnight fb oow gezeik fk lacht yess haten wit blauw tranen voet ruikt leggen ffe o_o gelukt toekomst huisje gelachen films ahahah iig aandoen slapenn been typen opzoeken bruin inleveren let knuffel dp sneller anderen make up schuld jaren gemeen rondje voldoende actie ehm ss sportschool band keel followers roos jeej smsen haalt aap gang verveling hm mevrouw minuut voeg afspraak energie moeders afsluiten enge gebroken stap make roze +gtst vrije 1000 alsjeblieft drank whahah plannen hondje pakt opdracht uitlaten +fml neit +gezellig spierpijn meter busje red az cont zitte las zoekt komende tekst enkel hahahaa melden gezeten heey uggs winter google kunne vaag vandaan tom lessen oeh wens kben kranten niels moee brief poetsen agenda party kaarten geweldige goodnight italie 23 tel x_x +ajax +pff nep veld kampioen kaas and kaart barca +fb hockey iedere chinees broodjes thomas kleur +doei +gm arme past onweer pff. heeeel mooiste aub 1 0 zojuist xp drama kleding perfect kwartiertje kus weltrusten tandenpoetsen beterschap totdat vak gisteravond prachtig duss nederlandse eraf gezellige vechten besteld chick kost miljoen liefst janken fotos makeup +zinin enso trekken drink neer genomen bril justin lijk +fail gun frankrijk cola leest zweer +honger kyken lokaal verteld kopje okeoke gooit +heerlijk meest haartjes planning midden rapport byna zwolle gezond belangrijk boot bloed vliegtuig hotel schreeuwen brand mannetje zorg p. kerk fan bas whapp hoeven real nicht aii gelopen springen mentor gaf rick beugel mense verdient +eten +linkdelieve drukke achja omgg knallen liet verslapen inhalen kusje lelijke laura kou un jahoor ikke triest ermee +np niett bro weertje vinger mar pik meedoen woordjes schoon achtergrond normale schijt with docent eitje strax we=ll maakte verschil geak ht antwoorden bedoelt trui groetjes ns project x. lichaam anne 2de verliezen sliep nope bam verkouden recht blond belde xbox reis gevraagd u_u minuutjes lisa tip stoel dak havo gaaf zware rijd jouu sterk jj mike lotte bekend hahhaha zovan all begon lijst magister robin mischien snappen um hekel win mongool heledag ofs gene vriendinnen leeft aangezien stuurt eem onvoldoende momenten finale jeetje soep swag lijken awh kevin cute gauw dann late walibi binnekort start oor ofniet wezen redelijk december daarvoor verbinding gepakt city vette buite 6e money gedoe schoool tyfus markt flink isgoed besef diep srry almere maja liefie sanne zee 5e stopt beeld leraren verzend mening boer hangt bart pf paard schieten ist yeaah brug lars checken 80 amper gn jee fock yeahh niveau aantal arnhem mooier draait tent doenn trug smsje happy brak filmpjes binne afmaken begrijpen bekijken volle lader verbrand vertrekken ongeluk retweeten vest schrok gaande verwijderd nummers biologie mw3 aangekomen oefenen bezoek random kfc zeiken jaah moete gat enigste zomervakantie +dutchteenagers kindjes waneer plus groeten medelijden donder appie oven dennis album hoofdstuk ter terecht at ahw gestopt uitleggen belgie c1000 beide klap +retweet uw eeh show eeen kwamen bos uiteten bedenken nf la leukste veelste vanmorgen betekend bday b1 bbq fouten jesse owh kome sam stinkt rekenen verleden ouder delen gewn hhaha examens fles okay jep toernooi +kut nah linken life media erge well anouk nest 200 vis aaaah keuze ex muur gereden ov waarvan duren uh c1 keuken beest mm goodmorning ft game veder april dubbel telkens schiphol techniek verhalen links metro sex vijf ne stemmen gefietst info steek one chocolade kim neej +love lenen zingt paardrijden negeren chocola konden maybe kaal jeroen aangesproken today fit getraind like schattie emmen wt bram honden trouwen optreden a1 mmm naaar bob gedacht wollah mega tegelijk omg. schoonmaken bestaan zenuwachtig ofzoo anderhalf vissen spelletje teen verwijderen meis kantine aflevering opgehaald wedstrijden groen zaak taal chil betere hahahahha zozo +feyenoord vooruit 2011 new leek verschrikkelijk hiero oortjes koffer holland veeel breng chagerijnig redden slet groene verdomme boom scoren krant schatjee schoolfeest kussen bakker paps hooor lied laden haast poep dochter a=dam jumbo ete eindhoven klok toetsweek keelpijn fleur collega hahahahahah hel probeert verdienen gevoelens mann wachte harder ooo bakkie id turkije blijkt hoesje +linkdefollower rook doel uhmm studio berichten muziekje vs wijn gitaar zucht vanzelf btg uiteindelijk theorie geregeld dankzij gozer mei koude gij amerika aardrijkskunde ondanks boekverslag dude boekje proefwerk groter rijdt demi verneukt zowat 2 0 fso fysio loveyou oren +lol zover know oei snoep opleiding carnaval terras ondertussen weekendje thnx koen down namen hoger tattoo uitslag gedronken heeeeel passen voelde ever cadeautje cc sokken lache telaat buurman cent downie humor morgenochtend eu eva zette baan kies jaman hartje och heee hakken beats jaaaaa cd faal goal omhoog alone nachtje bikini uitt gescoord vans bieber verkopen an whahahaha hahaah rijles tessa kweet geel avonds draai vandaar geschreven 35 londen vanwege contact vitesse afgesproken nagellak owja letterlijk gedroomd zieke nogal schrik ten alcohol interessant koekjes geil musical prijs bedoeld zonde sigaretje fris welterusten +koffie okey +3fm acht zaten pww 500 naomi glas ruik uhh goedd vroege leiden armen y parijs 27 lame mun aight cutie krullen centrum 60 raakt britt jwz drm seconden naa camera huiss plat geboren inmiddels spelers durft varen zoiezo discussie bushalte raak ervoor 26 pitte kwil twit 2013 jarige sluit vertraging chris derde scorro lullig 2 1 wats rtl video tentamen irriteren for +saai main schoot irriteer nina park omwt gesloopt schrijft ned nuu werke raad stal partij vreselijk dorst danny daagjes bedacht mascara min. breda hahahahahahaha gestolen wooow ikga officieel achterop tweetdeck schrijf engelse zooooo idk compleet ballen makkelijker weeet nouu sportdag veilig truste datum meld reet verven lullen do gebleven kindje schatt fietse winkelen >> gaaan trainer stuurde logisch opdrachten seconde hale e togh hahahhaha rijk gedachten zeiden knippen aangekleed dd neeeee seks leeftijd blind bullshit maccie rechts d8 feesten melk voorlopig gaten cadeau positief vertrouw negen opgeruimd roept visite wk reed negeer lakken maart tie wis peter ruben wassen +slajezelf ek ahhh +wtf aller w8 feit versie kaulo bouwen pesten smsjes wifey heh dinge friet pot verdiend nadenken paarden lenzen spongebob tje website twijfel doos dans true theo gepraat schelden ochtends ruim fake maag schooldag hulp raken juni roepen guy acc bereikbaar kutzooi lauw europa hell 7e ronaldo zinnen timeline jup voice samsung rijbewijs restaurant geeen wnr pyjama qqq ng overigens afscheid hoek zwanger mini chagrijnig popje frank nerd begrijpt file meegenomen uiteraard pad vingers keeper yesss strak papier +sweetdreams ckv klagen maa flauw pp plaatje +slaaplekker +chill 0000 vlieg i=m +tb gedownload paardje januari meids omin overleden dankjee kmoet jelle verzinnen enorm martijn tellen paul hoorn duitse enz. aleen verlaten bby mark ohhh pech pfffff reclames berichtje waauw trekt 300 etc. thijs piet bijzonder tv. omfg apple typ verschillende +wiestaateronder okeej paardjes geslagen pannekoeken vriendinnetjes badkamer +ziek +stage fissa kracht ciao +bye 333 aula steken gedachte spin ahahahaha stadion +bah herkansing maatje time os bepaalde etenn knuffelen savonds goedzo eej tennis prachtige uitkomen tweete jordy please standaard lekkerding muggen verstand allerliefste cafe 1 1 ontzettend zuipen julie wennen amber kyk niek verpesten vliegt vergaat nieteens update artikel apart tyd merel 28 mensjes fml vogel opweg poes +truste kansen enzoo oktober klappen deken september tour bo fitness vo verlies 06 xxxxx leukk breken speelde sirieus jeeej stellen scherm face dragen post sneu zacht zijn/haar ontvangen kutt halfuurtje keet doorlezen winkels mooizo teken heerenveen mug verwachten +rt koelkast balkon euhm dame wegg repetitie mattie geholpen reacties kelly beurt studie uitgaan hoppa pest steen appel augustus dusja opnemen prive baal slag ee test gedraag ivm hierzo weetniet doornemen bui bv leert ww efteling meerdere sterren doorgaan zwak thx ofnie hoge boel tong lieff hoogte schoen meegemaakt billen eg +pfff mol galaxy tsja amy college erachter speler bord +werk juli scheelt moooi romy this +leuk jazeker mailtje vera +eindelijk latijn +genieten sleutel jonguh perongeluk hemel spiegel banaan gedrag doorgaat opladen kaolo heus voorstellen 70 kater mailen wekkertje idioot gr spoort go optie energy what gebakken as heeerlijk verandert offline manon gelooft lieg liegt nadat mondeling just straf bar bereiken smaak haah boeie dichtbij vele gele peukie mpp sint tosti=s voorbeeld concert betaald tussenuren ware merken +heelherkenbaar t. chelsea hen goh mopp zonnen hik 3 0 ligge scoort oordopjes war kneus snapte tilburg ziekte black klik bedrijf gasten neger +penw cake ho grappen vi verdriet willem gedouched neuken ka drankje good benede simpel hans +faal verhuizen aids be emma kilo fcking jurkje uitgeslapen stoere gala werkdag nigger jongetje fzo +ff ste dj piercing naartoe overmorgen bedoeling pim beer ado messi whats lekkers abbo etc pro sneeuwt +sstnl actief concentratie alkmaar ronde aarde michelle fluiten belachelijk du erover middagje toevoegen not vecht awesome track pakke plaatsen ergste wachtwoord grieks ketting bijles wauww auw date karma malle bleef pfff. wild patrick marco spaans daphne gegooid voetballe oww hi ij mien worde gezelligheid joey verwijder waarmee zes coole huil ikea aaan echtt rekening schaam morgenavond gebelt +loveit maatschappijleer winnaar mogool ingepakt frietjes openen +nice voren lesje deden +koud kotsen drukte wii vlak basis basisschool jack hjb mc treinen kende automatisch klaarwakker rik euh bitches franse tips nablijven plassen zaal onthouden ervaring le +thuis lijn clip hmm. +blij mss halfuur pet serie melissa kroeg voortaan dushi gelegen vakken troep verdien moeilijke kapsalon naakt sander harry verkocht maarten tevreden regel julia zaken bil manege blok kutweer zonnebril vriendjes eredivisie wensen teminste henk xtje poepie lucky pen spontaan wedden qua steun yo +tienerthings ens day koekje mentions verandering +haat koe 90 uitzoeken ahahahah broekje briefje zodra tumblr harte wouter luistert voeren t/m get msn. maaike belle ooooh cm haaha tfoe hoeren barcelona werkstuk vervolgens goud commentaar talent interview gehouden centraal beginne kloten simon stelletje lll koudd esmee hijs faalt qq. mb slotje muts w8en gewend berg zuid ong anna gemeente beters kennis verliest bussen hema +kapot tweette android topper lippen zielige px appen vr rep pakje nikki bemoei +fijn why alleeen bruiloft nakomen joost tegenover tss daarmee daardoor gekker bankje sluiten ring sloeg david verstuurd sgool zoenen +werken download vwo bereik +zucht stijl ruimte verdrietig nieet arsenal konijn klaaar vertrek huilt awhh helemaaal traag schouder afkijken opzoek whahahah goeed 29 babe aankomen sophie opeten dats zagen spelletjes type behoorlijk vertelt waarop gaar trappen scheld stonden lunchen dieren beker waardoor heaa andersom followen engeland roy november vodafone style onderwerp mooiboy gezwommen dingetjes oef overleven nieuwjaar albert verzorging massage mart 3 1 ideeen raden boeiend tjaa single vooor doorheen fixen plekken duim vorm tieten hinderlijk poepen nachts schrikken absoluut onderzoek denise isa gras +topmoppen belangrijke bowlen aahw jeugd wonder strafwerk mo dekens amersfoort milan vlees portugal bakje winterjas don=t tekening +vakantie gedragen verander eah katten hartstikke stofzuigen gezocht kak +yeah gelyk lente daggie lekke loser job beloofd mi toegevoegd schooltje auto=s +weekend reizen dora verwarming mislukt +goedemorgen spul lunch vuur 05 leve +vt sterker e. deal big meent ii 08 zkr knappe verzamelen emo +balen gone blog zoe gemiddeld vry gesmst bedenk ikk jasper oorlog winst ft. geprobeerd ren yeaaah wi zing adidas piano eenmaal kado bewolkt oplossing adres broertjes mob apeldoorn afgepakt jb leip +goedbezig kansloos mtv mot collega=s diploma durven kosten bureau kinderachtig remco koel storing getrouwd doden +foodfromheaven det maan favoriet moois stilte w

Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting

Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting_checkout
  • Description
  • Shipping Info
  • Review
timberland online kopen Timberland Heren Timberland Earthkeepers , Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting timberland tassen,goedkoop kopen
premium full-grain en suède bovenkant van anLWG Silver-rated leerlooierij voor comfort en duurzaamheid
bovenste is DWR behandeld voor waterdichtheid te helpen voeten droog te houden
100% biologisch katoen veters
Extra-soft 100% gerecycleerd PET voering en voetbed dekking voor uitzonderlijk comfort
Anti-vermoeidheid uitneembaar voetbed biedt alle -dag comfort en ondersteuning
Duurzame zool is gemaakt met 15% gerecycleerd rubber

De Heren Earthkeepers ® Receptie Country Travel packs goed, rijdt goed en vooral stijlen ook. Een tijdloos Timberland® ontwerp dat gaat het hele jaar door werken voor u, met een kant-up voor en waterbestendig bovenste, de zachte binnenste en anti-vermoeidheid voetbed zorgt voor een superieure comfortabele pasvorm.

Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting
Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting
Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting
Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting
Beste Prijs Moss Suede Timberland Heren Earthkeepers Front Country Reizen DFH2G56QK48SYV,timberlandschoenen,timberland laarzen sale,geen belasting
You may also like...

Upcoming Events

xx/xx/xx:  Special Sale on all products from noon until 3:00 pm!

xx/xx/xx:  Mayor Bob will be on hand for the ribbon cutting ceremony marking the opening of our newest location!

xx/xx/xx:  More stuff!