Hi Linvi, comments at the bottom. Thanks
1.
Even at 90s, for some reason Twitter stops the stream occasionally before the 90 timeout has had a chance to recycle.
From what I understand you have a stream that stops working before the 90 first seconds it started? I am not sure I understand this clearly.
2.
The problem with this is that the code needs to be able to attempt to restart immediately rather than just wait for the next timer to elapse - otherwise it might miss events.
I agree but I cannot really see a solution to this. Twitter clearly states that a stream we can know that a stream is dead after 90 seconds without any information.
My worry is that if we specify a limit at 70 seconds for example, and the stream is not dead. At second 71 we are restarting the stream and a Tweet is added to the stream. We therefore missed the event. When the stream has restarted at second 72 we have missed it without knowing.
Do you think it would be a good idea to give the option to developers to themselves specify a timeout after which the stream will restart?
3.
Not only that, according to the documentation, it then needs to back off exponentially with any necessary retries if that connection is not successful for some reason.
The issue with the back off is that it is only available to user with elevated privileges on the Stream API. Do you think it is worth adding such a feature as so few developers will benefit from it?
(1). I soak tested for 24 hours. I had logging which logs started and stop events. I also made a note of the time that the start event began and checked it in the stop event. This allowed me to check for short sessions.
What I saw in normal use was a startsession event logged, followed 90s later by a stopsession event logged. In some cases though the stopsession fired before the 90 timer had fired which made me conclude that this was something else in the twitter API which was stopping the session early. The problem I had of course without adding more code to work around that was that the "anti stall" code was actually stalling and remained stalled until the rest of the 90s window expired to restart the session. Not nice. We are throwing away data.
(2). No I think you are right to enforce what the docs say. And if they say 90s then that should be it. The only problem is if the docs change at a later date then a method to override might be useful.
(3). I agree. It is not worth doing.
1.
Even at 90s, for some reason Twitter stops the stream occasionally before the 90 timeout has had a chance to recycle.
From what I understand you have a stream that stops working before the 90 first seconds it started? I am not sure I understand this clearly.
2.
The problem with this is that the code needs to be able to attempt to restart immediately rather than just wait for the next timer to elapse - otherwise it might miss events.
I agree but I cannot really see a solution to this. Twitter clearly states that a stream we can know that a stream is dead after 90 seconds without any information.
My worry is that if we specify a limit at 70 seconds for example, and the stream is not dead. At second 71 we are restarting the stream and a Tweet is added to the stream. We therefore missed the event. When the stream has restarted at second 72 we have missed it without knowing.
Do you think it would be a good idea to give the option to developers to themselves specify a timeout after which the stream will restart?
3.
Not only that, according to the documentation, it then needs to back off exponentially with any necessary retries if that connection is not successful for some reason.
The issue with the back off is that it is only available to user with elevated privileges on the Stream API. Do you think it is worth adding such a feature as so few developers will benefit from it?
(1). I soak tested for 24 hours. I had logging which logs started and stop events. I also made a note of the time that the start event began and checked it in the stop event. This allowed me to check for short sessions.
What I saw in normal use was a startsession event logged, followed 90s later by a stopsession event logged. In some cases though the stopsession fired before the 90 timer had fired which made me conclude that this was something else in the twitter API which was stopping the session early. The problem I had of course without adding more code to work around that was that the "anti stall" code was actually stalling and remained stalled until the rest of the 90s window expired to restart the session. Not nice. We are throwing away data.
(2). No I think you are right to enforce what the docs say. And if they say 90s then that should be it. The only problem is if the docs change at a later date then a method to override might be useful.
(3). I agree. It is not worth doing.