I haven’t made any changes to the fluff grabber since last year. Over that time, it’s been working to get past 200,000 (fluff)Books and it made it with ease. The cleaner, the thing that scans (fluff)Books to see if they’re working (not ban, not deleted, not private, etc) has worked it’s way to clean-put about 20,000Ã‚Â (fluff)Books. There have been an extraordinary number of clicks too,Ã‚Â 1,640,887 clicks to date (that’s more than 1 and half million clicks!)
Of course, it hasn’t been smooth sailing either. The crawler, the part of the system that makes the super-zoo grow by itself, isn’t healthy right now. I’m not sure exactly what happened to it, but here’s the explanation I have. It seems that the response time of (fluff)Friends on Facebook (they’re both slow, but together the effect is compounded) is much larger than the time-out-time that the system uses to fetch pages. So basically, the (fluff)Book takes 16 seconds to respond but the crawler times-out at about 9-10 seconds. Again, I don’t know if it’s the problem. I had added some debugging information to the crawler-ajax thread on the browse page. It only comes up when there is clearly an issue, which is just about all the time at this point.
How about fixing it? I’ll see if I can make the time-out-time longer, maybe to 20 seconds, but after that, if (fluff)Friends and Facebook themselves don’t get a bit speedier to respond, I’m not sure what’ll happen to the crawler.
Hope you guy’s have a good time fluffing with the Fluff Grabber!