This section presents an evaluation of our proposal on real data
collected from the Twitter microblogging platform. The datasets are
first presented in details. They are then used to analyze several
aspects of the learning procedure.

\subsection{Data Collection}

Twitter is the most popular microblogging platform. It allows any user
to send short messages which instantly become available to the users
who ``follow'' the message's author~\cite{TwitterSocialMedia}. Since the
service has been launched, in 2006, millions of users have been
writing more than 10 billions ``tweets'' (as ``messages'' are named in
Twitter). These messages can be comments about one's personal life,
can be interesting resources one wants to share with their followers,
but it can also be opinions on polemic topics. As explained earlier in
this paper, the real-time use of Twitter makes the latter category of
messages particularly valuable to learn the bias of a user. Other
noticeable uses of the instantaneous aspect of Twitter include the
ability to signal early warnings of
earthquakes~\cite{TwitterEarthquake}.

In this work, we consider two common topics that generate debate on
Twitter: politics and sports. The former category is represented by
the 2010 Brazilian Presidential Election and two sport championships
are processed: the 2010 Brazilian Soccer League and the 2010-2011 US
National Football League. Table~\ref{tab:summary} gives a general
overview of those three datasets. In all cases, the messages are
collected via the keyword-based Twitter API\footnote{available at
  http://apiwiki.twitter.com/}. In the Brazilian political context,
the names of the two main presidential candidates (Dilma Rousseff and
Jose Serra) are used. In the sportive contexts, the team names are
chosen keywords (the 12 most popular teams of the 2010 Brazilian
Soccer League; all 22 teams of the 2010 NFL Seasons). Notice, in Table~\ref{tab:summary}, that our
proposal runs very quickly; it only requires a few minutes to
process any of the three datasets, which represents months of
data. Indeed, each interval is processed (messages are grouped and bias is learned from the group) in less than a milisecond (see Table~\ref{tab:summary}). Given the time complexity of our approach, the
single core machine with 24GB of RAM memory we used in our experiments should handle applications with hundreds times more
messages per interval (if such applications exist).

\begin{table}
%  \setlength\abovecaptionskip{0pt}
%  \setlength\belowcaptionskip{6pt}
  \begin{footnotesize}
    \begin{center}
      \caption{General overview of the datasets.}
\begin{tabular}{|c|c|c|c|} \hline
    & \textbf{Brazilian Soccer League} & \textbf{NFL}  & \textbf{Brazilian Elections} \\\hline
  period &  2010-05-08 to 2010-12-15 & 2010-01-10 to 2011-02-17  & 2010-01-01 to 2010-12-17   \\\hline
  bias &  team preference & team preference  & candidate preference  \\\hline
  \# of classes &  12 & 22 &  2  \\\hline
  \# of tweets & 35,834,453   & 23,094,280 & 10,173,381   \\\hline
  \# of users  & 5,638,906   & 4,230,731 &  1,272,723  \\\hline
  avg. \# of tweets/users  & 6.4  & 5.5 & 8.0   \\\hline
  \# of known (labeled) users & 348,752 & 171,121 & 18,744 \\\hline
  \# of time intervals (minutes) & 365,986  & 262,541 &  397,428 \\\hline
  avg. \# tweets/interval & 98 & 88 & 26\\\hline
  execution time (seconds) & 215 & 94 & 48 \\\hline
  avg. processing time/interval (seconds) & 6.10$^{-4}$ & 3.10$^{-4}$ & 1.10$^{-4}$\\\hline
\end{tabular}
% \vspace {-5pt}
\label{tab:summary}
\end{center}
\end{footnotesize}
\end{table}

Our proposal learns biases in a supervised way. As a consequence, some
biases must be \emph{a priori} known. We decided to select users which
are clearly biased towards one class only. We considered three distinct
sources of evidence to generate ground truth of users:

\begin{enumerate}

\item Their use of specific
hashtags (hashtags are words that are emphasized in the sense that
Twitter use them to classify the message) helps their
identification. For example, users who post \emph{\#goJets} are likely
to support the New York Jets football team.

\item Some users explicitily unveil their team preference in the
description of their Twitter profiles, by mentioning the name
of their favorite team. 

\item In the two sportive
contexts, the number of times a user refers to a team (either with a
hashtag or with a simple mention) is counted. Whenever the most cited
team is three times higher than the sum of the mentions of all other
teams, the related user is supposed known and associated with a bias
that is the characteristic vector of the team. In the context of the Brazilian Political
Election, only hashtags, and not mentions, are considered. Indeed,
candidates are often mentioned in a negative way what makes this
information a very uncertain indication of a
support. 
\end{enumerate}



Users who do not fit in any of those three criteria are considered
unknown. Table~\ref{tab:summary} gives the number of known users in
each application.

The considered topics being highly dynamic, a 60 second time interval
is chosen. Figure~\ref{fig:cdf_peak_size} shows the cumulative
distribution of the number of messages per time interval. In all
datasets most time intervals contain very few messages: about 60\%
(Elections), 50\% (NFL Season) and 35\% (Brazilian Soccer League) of
the time intervals gather less than 10 tweets, whereas less than 5\%
of the intervals contain more than 1,000 messages. These intervals
relate to periods during which users are reacting to live
events. Interestingly, sport competitions drive more spikes of
activity than political discussions. This can be explained by the fact
that politic events are not commented live like sport
matches. Figure~\ref{fig:cdf_peak_size} clearly shows it: in sportive
contexts only, an inflexion of the curve separates the intervals
during live events from those outside these events. In contrast, most
of the political events are unplanned, therefore are less prone to be
commented live. Nevertheless, the following sections show that, even
in this context, biases can be learned from the temporal
locality. This locality is just more diffuse.

\begin{figure}
  \centering
  \includegraphics[width=0.6\textwidth]{figures/cdf_peak_size_BR2010}
  \caption{Cumulative distribution of the number of messages per
    minute. Whatever the dataset, most of the one-minute intervals
    contain very few messages.}
  \label{fig:cdf_peak_size}
 % \vspace{-0.4cm}
\end{figure}

Most microbloggers send very few messages but a few of them are very
active. This classical distribution is plotted in
Figure~\ref{fig:users} and explains why, in the remaining of this
section, the analysis is focused on the users who author less than a
few hundreds tweets. The subsequent sections evaluate the
applicability of our approach. All quality scores are the result of a
traditional 5-fold cross validation.

\begin{figure}
  \centering
  \includegraphics[width=0.6\textwidth]{figures/freq_user}
  \caption{Number of users w.r.t. the quantity of messages they send. Most of users send less than 10 messages over the period we analyzed, in all three datasets.}
  \label{fig:users}
\end{figure}

%\pcalais{MOSTRAR QUE A PRECISAO EH OBTIDA CONSIDERANDO APENAS X\% DOS MOMENTOS}
%\pcalais{VERIFICAR SE USUARIOS ESCOLHIDOS PRA GT TEM AS MESMAS CARACTERISTICAS DOS NAO-ESCOLHIDOS?}

\subsection{Precision versus Interval Size}

The only parameter of our algorithm is the size
of the temporal windows used to group messages. In Figure~\ref{fig:prec_interval}
we vary this size and evaluate its impact in accuracy. As we can see, impact
of interval size depends on the scenario, specially when we contrast sportive
and political scenarios: in the Brazilian Soccer League dataset, precision
decreases as the interval increases, this is, indeed, a result that shows
how temporal locality is high in this scenario. On the other hand, precision
for NFL supporters start decreasing after we consider intervals of 60 minutes,
what indicates that these users may have a different way of reacting during
games than the Brazilian soccer supporters. Finally, precision for Brazilian
Elections is somewhat independent of time interval, what is probably a result
of the fact that political comments are not driven by instantaneous real-life
events, but more long-term events such as a scandal about a candidate which
is debated during a whole week.

\begin{figure}
  \centering
  \includegraphics[width=0.52\textwidth]{figures/prec_interval_BR2010.pdf}
  \caption{Precision according to different interval sizes.}
  \label{fig:prec_interval}
 % \vspace{-0.4cm}
\end{figure}


\subsection{Precision versus Minimal Number of Messages in the
  Intervals}
\label{sec:min_size}

Let us first assess whether the intervals with little activity (\ie, a
few messages) makes it harder to learn the users'
biases. Figure~\ref{fig:peak_size_precision_recall} plots the
precision (given a user, only the class that dominates her bias is
considered) and the recall (\ie, the number of users whose biases are
learned) obtained when more and more of the least active intervals are
discarded (the x axis gives the minimal number of messages an interval
must contain to be considered). Equations 1 and 2 detail how precision
and recall are calculated.

\begin{equation}
\label{precision}
\mbox{Precision} = \frac{
			                    |\{\mbox{users whose class has been correctly predicted}\}|
			                  }
			                  {
			                    |\{\mbox{users who have posted at least once during the intervals considered}\}|
			                  }
\end{equation}

\begin{equation}
\label{recall}
\mbox{Recall} = \frac{
			                    |\{\mbox{users who have posted at least once during the intervals considered}\}|
			                  }
			                  {
			                    |\{\mbox{total number of users in the dataset}\}|
			                  }
\end{equation}




 Of course, the recall monotonically
decreases when we require more activity in order
to consider a interval, as less data is considered as we increase this threshold. The precision first increases
and then decreases. Back to Figure~\ref{fig:cdf_peak_size}, it appears
that, the maximum is obtained when only the top 20\% (even less for
the NFL data) of the intervals (w.r.t. the number of messages they
contain) is considered. This confirms the supposed reason for the
observed temporal locality. Users express their bias in reaction to
live events. When no such an event happens (or is about to happen when
the event is planned), considering the related time periods makes it
harder to learn the biases. Note that simultaneous events can drive
users from different viewpoints to manifest at the same time; those
multiple events will help on discriminating those viewpoints from 
the viewpoints which are currently not related to those events. In general, however,
most events should occur in isolation, for example, most of goals and touchdowns
do not occur at the same time even if multiple matches are taking place simultaneously.  

Let us emphasize, however, that our proposal does not \emph{require}
to fix such a minimal number of messages per time interval. Indeed, in
Figure~\ref{fig:peak_size_precision_recall}, the precisions at 0 are
not much different from those at the maximums of the curves. The
achieved precisions are far above those of a random learning
($\frac{1}{|\mathcal{C}|}$). Almost 55\% (vs. 8.3\% for a random
learning) of the soccer supporters, 45\% (vs. 4.5\%) of the NFL
self-made commentators and 75\% (vs. 50\%) of the users interested in
Brazilian politics have their dominant bias that are correctly learned
by our proposal.

% There are two reasons explaining the importance of the low-activity
% time intervals. First of all, every event that catches a lot of
% attention is still commented a long time after its occurrence. When
% such an event is planned (\eg, a sport game), microbloggers interested
% in it also give their views before. For example, if two teams A and B
% will play a match on the weekend, and team C will not, supporters of
% the teams A and B discuss it all the weekend long (giving their
% expectations about it), whereas supporters of the team C show less
% activity. Thus, even time intervals at late night are useful. Users
% that simultaneously express themselves at that time probably keep on
% commenting evening news or anticipate an event that will take place
% the day after.

% Furthermore, users who ``retweet'' each other, \ie, who repeat each
% other's messages, support each other's views: they have similar
% biases. It turns out that retweets usually are sent right after the
% original message is read (in the so-called ``timeline''). In fact,
% \cite{TwitterSocialMedia} has recently shown that about 5\% of the
% retweets occur less than one minute after the original message. This
% temporal locality of the retweets is an additional argument in favor
% of clustering similar users according to their temporal patterns. It
% also partially explains why low-activity periods, where no live event
% is commented, carry information that is relevant to learn users'
% biases.

%Note,
%however, that our algorithm weights
%intervals in proportion to the level
%of activity they exhibit.

\begin{figure}
  \centering


  \subfloat[Brazilian 2010 Soccer League] {\label{fig:BR2010_prec_recall}\includegraphics[width=0.52\textwidth]{figures/peak_size_precision_recall_BR2010.pdf}}
  \subfloat[NFL 2010-2011 Season] {\label{fig:NFL_prec_recall}\includegraphics[width=0.52\textwidth]{figures/peak_size_precision_recall_NFL.pdf}} 
  
  \subfloat[Brazilian Elections] {\label{fig:ELEICOES_prec_recall}\includegraphics[width=0.52\textwidth]{figures/peak_size_precision_recall_ELEICOES.pdf}}

  \caption{Recall and precision w.r.t. the minimal number of messages
    a time interval must contain to be considered in this
    experiment. The recall decreases when less data is considered. The
    precision reaches a maximum when the periods of least activity are
    ignored.}
  \label{fig:peak_size_precision_recall}
 % \vspace{-0.4cm}
\end{figure}


\subsection{Precision versus User Frequency}

Another aspect that certainly affects the ability to automatically
learn a given users' bias is how often she expresses herself on the
studied topic. Figure~\ref{fig:prec_supp} shows the precision of the
bias w.r.t. the number of messages that are available to learn this
bias. Each point in the plotted curves corresponds the average
precision the users having sent, in the considered data, the number of
tweets defining the abscissa.

\begin{figure}
  \centering
  \subfloat[Brazilian 2010 Soccer League] {\label{fig:BR2010}\includegraphics[width=0.52\textwidth]{figures/prec_supp_BR2010}}
  \subfloat[NFL 2010-2011 Season] {\label{fig:NFL}\includegraphics[width=0.52\textwidth]{figures/prec_supp_NFL}}
  
  \subfloat[Brazilian Elections] {\label{fig:Elections}\includegraphics[width=0.52\textwidth]{figures/prec_supp_ELEICOES}}

  \caption{Precision of the normalized biases w.r.t. to the number of
    messages available to learn it. In Sport contexts, our proposed technique is significantly better than a majority-class classifier. In the political context, usefulness of temporal information in detecting users' bias is less evident, although it is still better than a majority-class classifier}
  \label{fig:prec_supp}
 % \vspace{-0.4cm}
\end{figure}

Of course, the more messages are sent, the better the user's bias is
learned. Nevertheless, in the sport contexts, even when one single
message is available, the user bias is learned in a way that is
significantly higher than the result of a majority-class classifier, i.e., a classifier which always guess the most frequent class. In the
political context, it is remarkable that, although political comments
on Twitter do not generate significant peaks of activity (see
Figure~\ref{fig:cdf_peak_size}), biases can be learned from the
temporal proximity between users' posts. We believe that the events,
in reaction to which the users express their biases, are more diffuse
than those of sport (\eg, goals). For example, politically biased
users are keen to send messages 1) when a scandal involving the
political opponent is revealed, 2) when their candidate is on TV
presenting her/his proposals, or 3) during a debate that turns in
favor of her candidate of choice. As a consequence, longer time
intervals may improve the results some more.

% In the sport contexts, it is interesting to notice that the precisions
% are lower when only the most vocal users are considered. For example,
% the users who have sent more than 300 messages to comment the NFL have
% their biases predicted with a 60\% precision. This is significantly
% lower than the 68\% precision obtained for all users with at least 50
% tweets. This phenomenon occurs because these users do not only comment
% the events related to their favorite team. Interestingly, no such
% phenomenon occurs with Brazilian political discussions: the more vocal
% a user, the easier it is to learn her bias. The reason for that may be
% the polarized aspect of this application. Because only two opposite
% classes are considered, there usually is only one event that is
% discussed at a time and this event is either at the advantage of a
% candidate or at the advantage of the other. On the opposite, sport
% scenarios involve a variety of independent matches that simultaneously
% take place.

%\subsection{Precision versus Training Size}

%In this Section we analyze the effect of two
%types of information which are required for
%our classification algorithm: number of
%users of known class and number of time intervals
%considered for classification. On both experiments,
%we sampled users and intervals and evaluated
%precision.

%\pcalais{TENHO QUE TERMINAR ESTA SUBSECAO.}

%\begin{figure}[t]
%  \centering
   
%  \subfloat[Brazilian 2010 Soccer League] {\label{fig:BR2010}\includegraphics[width=0.5\textwidth]{figures/user_pct.pdf}}
%  \subfloat[NFL 2010-2011 Season] {\label{fig:NFL}\includegraphics[width=0.5\textwidth]{figures/user_pct.pdf}}
%  \subfloat[NFL 2010-2011 Season] {\label{fig:NFL}\includegraphics[width=0.5\textwidth]{figures/user_pct_ELEICOES.pdf}}
%  \caption{Precision increases as more users of known bias are sampled.}
%  \label{fig:user_pct}
 % \vspace{-0.4cm}
%\end{figure}


%\begin{figure}[t]
%  \centering

%  \subfloat[Brazilian 2010 Soccer League] {\label{fig:BR2010}\includegraphics[width=0.5\textwidth]{figures/peak_pct.pdf}}
% \subfloat[NFL 2010-2011 Season] {\label{fig:NFL}\includegraphics[width=0.5\textwidth]{figures/peak_pct.pdf}}
%  \subfloat[Brazilian Elections] {\label{fig:NFL}\includegraphics[width=0.5\textwidth]{figures/peak_pct_ELEICOES.pdf}}
%   \caption{Precision increases as more time periods are sampled.}
%  \label{fig:peak_pct}
 % \vspace{-0.4cm}
%\end{figure}


\subsection{Monitoring Events in Real-Time Using User Bias Knowledge}

In this subsection, our proposal is shown to be valuable to understand
the evolution of Twitter's bias as a
whole. Figure~\ref{fig:Cruzeiro_Atletico_match} shows the number of
supporters of the teams Cruzeiro and Atletico Mineiro, posting during
the Brazilian Soccer match which clashed them on October 24, 2010. The
number of messages increases as soon as the match begins at
time=150. At minute=200, Atletico Mineiro was already winning by 3-0,
and, indeed, three spikes are observable in the interval [150,200].
At those spikes, comments from supporters of Atletico Mineiro are at
least twice more frequent than those from Cruzeiro's supporters. The
reason is simple: users celebrate the goals of their favorite teams
more than they comment on the failure of its defense. Thus, monitoring
the bias of who is expressing itself on Twitter allows to infer to
whom the event is in favor of. In some way, this can be understood as
a group sentiment analysis.

\begin{figure}
  \centering
  \includegraphics[width=0.6\textwidth]{figures/classico.pdf}
  \caption{Cruzeiro 3 -- 4 Atletico Mineiro match. When a team scores,
    its supporters send a lot more messages than the supporters of the
    opposite team. After Atletico Mineiro's win, at 270', its
    supporters celebrate even more intensively. }
  \label{fig:Cruzeiro_Atletico_match}
 % \vspace{-0.4cm}
\end{figure}

We used our proposal to follow Twitter's bias along the Pittsburgh
Steelers vs. Green Bay Packers NFL match. Figure~\ref{fig:superbowl}
not only plots the number of tweets sent by supporters of each of the
two competitors, but also the number of messages authored by users
biased towards other teams. During this match, most of the comments
came from Steelers fans. At the 260$^{\text{th}}$ minute, when the
game ended and the Packers became NFL Champions, an activity spike
occurs. Although there were, during the match, a minority of Packers
supporters, they dominate this spike.

\begin{figure}
  \centering
  \includegraphics[width=0.6\textwidth]{figures/superbowl.pdf}
  \caption{SuperBowl match between Pittsburgh Steelers and Green Bay
    Packers, won 31--25 by the Packers. Packers supporters react more
    intensively once the match comes to an end, \ie, after the team is
    proclaimed NFL Champions. The match also caught the attention of
    users biased towards other NFL teams.}
  \label{fig:superbowl}
 % \vspace{-0.4cm}
\end{figure}

Following the dynamics of Twitter's bias during events is a
straight-forward application of our approach. As users' bias are
learned from their previous messages, this analysis can be performed
``online'' (\ie, as the events occur and the user biases are
learned). In other terms, as long as a ``buzz'' is associated with a
class, our proposal can catch it while it occurs and in a
content-independent way.


