Commented Issue: SearchTweetsRecursively breaks prematurely [2138]

Hello!

Let me first congratulate you on a fantastic job. The framework has some brilliant uses.

I noticed that when searching for tweets greater than the max (>100), there are varied results. I will see no where near my input amount, despite it being well under the rate limit (i've tried 200 - 10,000) and the number of results seem to be different each time.

I stepped through the code and think I have found the issue. There is an if statement in the below method which breaks once the returned result is less than the count parameter you supplied. Now I can see the reason for this - presumably if the count is less, then the rate limit has been hit! But, I have found this is not always the case. Sometimes the Twitter API can be a little intermittent and return 99 results instead of my 100 max, in which case the method breaks and there are still plenty of tweets which could be returned!

In fact, if you comment out that code altogether, I was seeing 10,000 results returned instead of the 1000 max previously.

The code potentiallly needs to be altered so that the method breaks only if the rate limit has actually been hit. Simply commenting out is not enough because an exception is thrown when trying to parse an empty HTTP result set.

Many thanks,
keep up the great work,

James.

private IEnumerable<ITweetDTO> SearchTweetsRecursively(ITweetSearchParameters tweetSearchParameters)
{
var searchParameter = CloneTweetSearchParameters(tweetSearchParameters);
searchParameter.MaximumNumberOfResults = Math.Min(searchParameter.MaximumNumberOfResults, 100);

string httpQuery = _searchQueryGenerator.GetSearchTweetsQuery(searchParameter);
var currentResult = GetTweetDTOsFromSearch(httpQuery);
List<ITweetDTO> result = currentResult;

while (result.Count < tweetSearchParameters.MaximumNumberOfResults)
{
var oldestTweetId = GetOldestTweetId(currentResult);
searchParameter.MaxId = oldestTweetId;
searchParameter.MaximumNumberOfResults = Math.Min(tweetSearchParameters.MaximumNumberOfResults - result.Count, 100);
httpQuery = _searchQueryGenerator.GetSearchTweetsQuery(searchParameter);
currentResult = GetTweetDTOsFromSearch(httpQuery);
result.AddRange(currentResult);

if (currentResult.Count < searchParameter.MaximumNumberOfResults)
{
// There is no other result
break;
}
}

return result;
}
Comments: ** Comment from web user: jmsmcniff **

Thanks Linvi.

Perhaps you could change the if statement to read something like this:

if (currentResult.Count == 0 )
{
// There is no other result
break;
}

This would break only if no results were returned from the API.

Or, you could build in a quick call over to the rate limit API before you enter the while loop to return the number of results left, multiply by the number of results possible per API call (100 for the search API) then have your own counter within the loop which decreases each time results are returned from the API.

This is the kind of logic I am using in my own code which enables me to bulk download Users and Tweets, with pauses inbetween the rate limits:

var remaining = RateLimit.GetCurrentCredentialsRateLimits().SearchTweetsLimit.Remaining * MaxNumberOfResultsPerCall;

while (result.Count < tweetSearchParameters.MaximumNumberOfResults)
.......
currentResult = GetTweetDTOsFromSearch(httpQuery);
remaining -= currentResult.Count();

if (remaining == 0 )
{
// There is no other result
break;
}

I completely agree with your point about wanting to reduce the amount of parameter choices for the end user. The main issue I see is that the number of results from Twitter aren't always what you expect. From looking through their discussions, I get the impression the Max is more of an "up to" rather than a guaranteed amount.

Thanks,
James.