Question about (Internet) spiders

[Oniya]

Every so often, I see a spider 'looking' through a thread in the 'Who's Online' screen.  Does that 'look' add to the views tally on the threads, or does the forum filter out views from search engines?

[Beguile's Mistress]

That's a good question, Oniya.  I've wondered about that from time to time myself.

[Vekseid]

It's current configured to not count search engine spider hits for overall forum views, though I don't know if that counts on the per-thread level. Lessee...

Code Select Expand


        if (empty($_SESSION['last_read_topic']) || $_SESSION['last_read_topic'] != $topic)
        {
                $smcFunc['db_query']('', '
                        UPDATE {db_prefix}topics
                        SET num_views = num_views + 1
                        WHERE id_topic = {int:current_topic}',
                        array(
                                'current_topic' => $topic,
                        )
                );

                $_SESSION['last_read_topic'] = $topic;
        }


So basically it's going to increment almost every time a spider hits it.

[Oniya]

So if I put a post in the public (searchable) area, it could get hit by 40 spiders (I've seen more than that online at once) and not a single human, and it would show 41 views (counting mine)?

[Vekseid]

Well, most of them are just downloading in parallel. There are only a handful of spiders that make a serious attempt at keeping up-to-date on everything, and most of it is governed by the sitemap - Googlebot, Slurp (Yahoo), Bingbot, and any other spider that understands sitemaps should only visit it again if it's been updated in some manner.

That doesn't always happen, but the dumber spiders do get hooked on sessions so they won't necessarily get counted a bajillion times.

[Caeli]

How interesting.

You know, I've also seen spiders "posting" to a thread (or at least, that's what they're listed as doing). Obviously, they don't post, so why exactly does that message show up?

[Vekseid]

The action description you see is triggered by how SMF understands the URL of whomever is visiting it.

Open up another browser where you aren't logged into Elliquiy. Copy the post link to a thread where you are logged in, paste it into the unlogged-in browser.

Now go back to your logged-in browser, go to 'who's online', and view guests. You're non-logged-in self is 'posting'.

[Caeli]

Oh, how fascinating!

These spiders, they intrigue me. >_>

[Gabby girl]

wait, what's a spider, and are they a bad thing here?   They don't bite, do they?  hehe    j/k on that last part :p

[Oniya]

Spiders (in the Internet sense) are little programs that search engines use to compile web pages for their search results.  As spiders help to steer people towards pages that match what they're looking for, spiders are actually rather beneficial.  However, as anything in the public area of Elliquiy can be 'seen' by spiders (and therefore, can be searched for), they are yet another reason to keep private stuff (like contact info) out of the public area.

[Beguile's Mistress]

I was pointed to this article a while back by one of my friends when I asked her about web crawlers.