
- Deadline: 29-AUG-2017

*******************************************************************************************************************
Reviewer A:

Summary of main issues to be addressed:

1) clarify the problem, adequately contextualizing it in the literature
The formulation of the problem, namely if it is not simply the cold start problem.
Concerning the definition of the problem, in the letter submitted with the
paper, the authors distinguish the “ramp-up” and “cold start”
problems. I think I can summarize their explanation as follows: the ramp-up
problem is when you have no ratings at all while, in the cold start problem
there are some, although few ratings. As the authors acknowledge, the
problems are highly related. In the recommender systems literature, as far
as I know, the “ramp-up problem” term is not used but the term “new
user problem” is. See, for instance, the following survey, which is cited
in your paper:

Bobadilla, J., Ortega, F., Hernando, a., & Gutiérrez, a. (2013).
Recommender Systems Survey. Knowledge-Based Systems, 46, 109–132.
https://doi.org/10.1016/j.knosys.2013.03.012

I still think this needs to be more deeply discussed in the paper. As I
mentioned in my previous review,  a mathematical formalization of the
problem would probably be very helpful. In any case, I will make a
suggestion. If you take the perspective of:

A. M. Rashid, I. Albert, D. Cosley, S. K. Lam, S. M. McNee, J. A. Konstan,
and J. Riedl, “Getting to know you: learning new user preferences in
recommender systems,” in Proceedings of the 7th international conference
on Intelligent user interfaces, pp. 127–134, ACM, 2002.

which is cited in the paper, I think you could present your work as
supporting a variant of the strategies for item selection: minimize user
effort and maximize recommendation accuracy, focusing on niche users.

=> descrever melhor o ramp-up x cold-start
=> Respostas: DONE!

Na introdução: This problem is known in the literature in two ways: (1) Cold-Start problem; and (2) Ramp-up problem. The Cold-Start problem is related to generating recommendations for new users, whose consumption history is small and little relevant \cite{adomavicius2005toward, bobadilla2013recommender}. On the other hand, the Ramp-up problem is even more complicated, since it is related to first-time users, for whom there is still no information in the system \cite{martin2014}.

No related work: The literature in Recommender Systems defines this problem as: (1) Cold-Start problem; and (2) Ramp-up problem. The Ramp-up problem is commonly deemed as a variation of the Cold-Start one~\cite{bobadilla2013recommender}. Despite being closely related, both problems should be addressed differently. Whereas the Cold-Start problem deals with users with small consumption histories (i.e., inactive or new users), in the Ramp-up there is no consumption information about the users (i.e., first-time users). For e-commerce systems, any information is better than none and, for this reason, Ramp-up is a major challenge for Recommender Systems.

------------------------------------------------------------------------------------------------------------

2) Concerning the section on related work:
-in the related work, the authors categorize papers in three classes.
Isn’t there a taxonomy of methods that can be reused?
-the argument against questionnaires is not very convincing, as you may
simply ask: list 5 items you like and 5 times you don’t
-the argument in favor of non-personalized RS is a bit contradictory with
the motivation of this work (namely “generalization capability” and
“good performance”).

=> melhorar esses tópicos. TO DO!
=> Resposta: argumentar com os revisores!!!

------------------------------------------------------------------------------------------------------------

3) improve the explanation of the method:
The explanation of the method could be significantly improved. First of all,
the Maximum Coverage method should be explained when it is introduced, even
if in an informal way. Furthermore, there are several smaller issues,
including:
-in page 4 it is stated that “The problem is to determined the subset
F_k^* of size k…” It’s a subset of which set?
-Algorithm 1 is confusing, especially line 4
-the caption of fig.1 doesn’t seem appropriate.
An additional issue concerning the method is the computational complexity,
which is cubic. This should not be claimed as a scalable strategy. In fact,
execution time results should be included.

=> melhorar a explicação do MaxCov e do algoritmo
=> Resposta: DONE!

In section 3, we add a formaly description and explain better the algorithm.

However, observing the greedy algorithm presented in Algorithm 1, it is possible to
realize that the chosen items are the ones that maximize the number of users at each iteration. In
other words, the covered users are excluded from the set R, which corresponds to the analyzed users
(line 6), so that the S item that has not been previously selected (S ∈ F \ F ∗ ) is selected and has the
largest intersection with uncovered users (|S ∩ R|), as shown in line 4. Consequently, as the number of
iterations approximates to k, the selected items become less popular.

------------------------------------------------------------------------------------------------------------

4) formalize all the measures correctly and motive why they are used
Concerning the description of the evaluation measures, accuracy is defined
in a way which is, according to Bobadilla, J., Ortega, F., Hernando, a., &
Gutiérrez, a. (2013) a set recommendation metric.
Then, the explanation of diversity is not clear as well.

=> melhorar as descrições das métricas de avaliação
=> Resposta: DONE!

On the other hand, $diversity$ metric generally applies to a set of items, and relates to how different items are when compared with each other \cite{ricci2011introduction}. Basically, it is calculated as the complement of Pearson’s similarity of the
recommended items $R$, as shown by the equation \ref{div}. This metric comes from the framework proposed by \cite{vargas2011rank}. This metric represents the diversity of items based on the expected average distance from an item to an item list (ILD).

------------------------------------------------------------------------------------------------------------

5) motivate the new measures adequately and contextualize it in the literature
The F-measure is a well established measure in information
retrieval. It’s not clear why a new F-measure is proposed here. It also
seems related to the “balanced strategies” in A. M. Rashid, I. Albert,
D. Cosley, S. K. Lam, S. M. McNee, J. A. Konstan, and J. Riedl (2002).

=> escrever uma justificativa plausível e referenciada (difícil) para o New F-measure
=> Resposta: DONE!

we add:

Finally, we propose and use a new metric in this work that aims to measure the harmonic mean between accuracy and diversity. This metric has the objective to consider the trade-off between diversity and accuracy, just to evaluate a recommender systems based on both criteria. In order to do it, we normalize the accuracy values considering its highest value according to parameter $k$. High values indicate that the recommender is able to present a useful and diverse set of items to the users. The results of this metric make clear the need for recommender to achieve both accuracy and diversity.

------------------------------------------------------------------------------------------------------------

6) explain research questions more carefully
The research questions investigated in each part of the study should be
explained more carefully in the introduction of section 5.

=> alterar no texto
=> Resposta: DONE!

In introduction of section 5, we add:

In this section, we discuss the main results of applying the proposed evaluation methodology to first-
time users selected in MovieLens 1M and 10M datasets. Initially, we perform a comparison and
analysis of the different recommendation strategies, considering the set of items recommended. For
this, we propose to evaluate the intersection of the items recommended by each strategy using a Venn
diagram. Next, we evaluate the utility and diversity of each strategy to evaluate the performance of
our Maximum Coverage strategy. Our objective is to evaluate the quality of the recommendations of
each of the strategies considering the quality dimensions. Finally, we perform specific analyses of the
strategies, considering a real scenario in which a maximum of twenty items are recommended. In this
step, we specifically evaluated the recommendation of up to twenty items, such as a real e-commerce
system such as Amazon or Netflix, which has a maximum of 20 items at a time for users.

------------------------------------------------------------------------------------------------------------

7) extend the experimental setup by testing more algorithms or analyzing the robustness of the method by manipulating the dataset used

=> adicionar o random popularity
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------

8) compare the proposed method with a method that recommends items for the masses with a small number of random items

=> adicionar o random popularity
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------

9) include comparison in terms of execution time

=> fazer uma tabela do tempo gasto para recomendar 5, 10 e 20 itens => Diego Carvalho.
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------

10) It is not clear why the scenarios of 5, 10 and 20 are realistic and th others aren’t.

=> Explicar isso melhor.
=> Respostas: DONE!

*******************************************************************************************************************
Reviewer B:


1) The paper presents a non-personalized approach for movie recommendation
based on Maximum Coverage. The approach is interesting and the results shows
competitive results with respect to those based on popularity and others. It
would have been important to compare (or justify why it is not compared)
with the approaches described in Section 2.1.

=> justificar isso.
=> Respostas: Argumentar com os revisores.

------------------------------------------------------------------------------------------------------------

2) Also, in many graphics the approach performs a little worse than others (like Fig 4 and some of Fig 7).
The reasons for this should be further explained and justified. 

=> Não entendi essa questão.
=> Respostas:

------------------------------------------------------------------------------------------------------------

3) More importantly, in many cases that differences in some metrics are small. The
statistical significance of differences should be included in the analysis of results. 

=> Adicionar alguns testes estatísticos a mais.
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------

4) Finally, conclusions should be more compelling.

=> melhorar isso.

------------------------------------------------------------------------------------------------------------

5) Regarding presentation, there are several things to correct. Section 2.1 is
a unique subsection in Sec 2, the text should be split in at least 2 subsections. 

=> Fazer isso.
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------

6) In the end of Sec 1 the organization of the work should be included. 

=> Incluir essa subseção.
=> Resposta> DONE!

------------------------------------------------------------------------------------------------------------

7) The quality of Fig 1 is poor. 

=> Melhorar a qualidade das imagens.

------------------------------------------------------------------------------------------------------------

8) There are a number of errors to correct, for example:
In last paragraph of sec 2.1: “consider aims”
Sec 5, 1st paragraph: “we perform” --> “we performed”

=> Corrigir.
=> Respostas: DONE!

------------------------------------------------------------------------------------------------------------
