Commented Unassigned: SearchTweetsRecursively breaks prematurely [2138]

Hello!

Let me first congratulate you on a fantastic job. The framework has some brilliant uses.

I noticed that when searching for tweets greater than the max (>100), there are varied results. I will see no where near my input amount, despite it being well under the rate limit (i've tried 200 - 10,000) and the number of results seem to be different each time.

I stepped through the code and think I have found the issue. There is an if statement in the below method which breaks once the returned result is less than the count parameter you supplied. Now I can see the reason for this - presumably if the count is less, then the rate limit has been hit! But, I have found this is not always the case. Sometimes the Twitter API can be a little intermittent and return 99 results instead of my 100 max, in which case the method breaks and there are still plenty of tweets which could be returned!

In fact, if you comment out that code altogether, I was seeing 10,000 results returned instead of the 1000 max previously.

The code potentiallly needs to be altered so that the method breaks only if the rate limit has actually been hit. Simply commenting out is not enough because an exception is thrown when trying to parse an empty HTTP result set.

Many thanks,
keep up the great work,

James.

private IEnumerable<ITweetDTO> SearchTweetsRecursively(ITweetSearchParameters tweetSearchParameters)
{
var searchParameter = CloneTweetSearchParameters(tweetSearchParameters);
searchParameter.MaximumNumberOfResults = Math.Min(searchParameter.MaximumNumberOfResults, 100);

string httpQuery = _searchQueryGenerator.GetSearchTweetsQuery(searchParameter);
var currentResult = GetTweetDTOsFromSearch(httpQuery);
List<ITweetDTO> result = currentResult;

while (result.Count < tweetSearchParameters.MaximumNumberOfResults)
{
var oldestTweetId = GetOldestTweetId(currentResult);
searchParameter.MaxId = oldestTweetId;
searchParameter.MaximumNumberOfResults = Math.Min(tweetSearchParameters.MaximumNumberOfResults - result.Count, 100);
httpQuery = _searchQueryGenerator.GetSearchTweetsQuery(searchParameter);
currentResult = GetTweetDTOsFromSearch(httpQuery);
result.AddRange(currentResult);

if (currentResult.Count < searchParameter.MaximumNumberOfResults)
{
// There is no other result
break;
}
}

return result;
}
Comments: ** Comment from web user: linvi **

Hi,

I already had the knowledge of this "issue". I was assuming that the results sent from Twitter were always correct (which does not seem to be true).

Therefore, I have 2 choices keep considering that Twitter is supposed to be a trustful source or allow the users to force an operation to continue. I am quite uncertain of the choice I have to make on this.

I might introduce an additional parameter to the search query parameters leave the choice to the users. I don't know what I will do.

Please give me your point of view on the subject.
Linvi