You are either not logged in or not registered with our community. Click here to register.
 
December 09, 2016, 07:07:56 PM

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

Click here if you are having problems.
Default Wide Screen Beige Lilac Rainbow Black & Blue October Send us your theme!

Hark!  The Herald!
Holiday Issue 2016

Wiki Blogs Dicebot

Author Topic: Question about (Internet) spiders  (Read 776 times)

0 Members and 1 Guest are viewing this topic.

Offline OniyaTopic starter

  • StoreHouse of Useless Trivia
  • Oracle
  • Carnite
  • *
  • Join Date: Sep 2008
  • Location: Just bouncing through. Hi! City of Roses, Pennsylvania
  • Gender: Female
  • One bad Motokifuka. Also cute and FLUFFY!
  • My Role Play Preferences
  • View My Rolls
  • Referrals: 3
Question about (Internet) spiders
« on: February 02, 2012, 03:54:26 PM »
Every so often, I see a spider 'looking' through a thread in the 'Who's Online' screen.  Does that 'look' add to the views tally on the threads, or does the forum filter out views from search engines?


Offline Beguile's Mistress

  • Time flies like an arrow ~ Fruit flies like a banana ~ Elliquiy's Fair-E Godmother
  • Dame
  • Carnite
  • *
  • Join Date: Jul 2009
  • Location: Faeleacanvald ~ The Steeler Nation ~ Home of Lord Stanley's Cup 2016 ~ She won't stay throwed! ~ 48\22-5\1\11-5\7
  • Gender: Female
  • Perpetual Notion Machine ~ 'What if...?'
  • My Role Play Preferences
  • View My Rolls
  • Referrals: 3
Re: Question about (Internet) spiders
« Reply #1 on: February 02, 2012, 03:58:01 PM »
That's a good question, Oniya.  I've wondered about that from time to time myself.

Online Vekseid

Re: Question about (Internet) spiders
« Reply #2 on: February 02, 2012, 04:02:56 PM »
It's current configured to not count search engine spider hits for overall forum views, though I don't know if that counts on the per-thread level. Lessee...

Code: [Select]
        if (empty($_SESSION['last_read_topic']) || $_SESSION['last_read_topic'] != $topic)
        {
                $smcFunc['db_query']('', '
                        UPDATE {db_prefix}topics
                        SET num_views = num_views + 1
                        WHERE id_topic = {int:current_topic}',
                        array(
                                'current_topic' => $topic,
                        )
                );

                $_SESSION['last_read_topic'] = $topic;
        }

So basically it's going to increment almost every time a spider hits it.

Offline OniyaTopic starter

  • StoreHouse of Useless Trivia
  • Oracle
  • Carnite
  • *
  • Join Date: Sep 2008
  • Location: Just bouncing through. Hi! City of Roses, Pennsylvania
  • Gender: Female
  • One bad Motokifuka. Also cute and FLUFFY!
  • My Role Play Preferences
  • View My Rolls
  • Referrals: 3
Re: Question about (Internet) spiders
« Reply #3 on: February 02, 2012, 04:06:18 PM »
So if I put a post in the public (searchable) area, it could get hit by 40 spiders (I've seen more than that online at once) and not a single human, and it would show 41 views (counting mine)?

Online Vekseid

Re: Question about (Internet) spiders
« Reply #4 on: February 02, 2012, 04:13:23 PM »
Well, most of them are just downloading in parallel. There are only a handful of spiders that make a serious attempt at keeping up-to-date on everything, and most of it is governed by the sitemap - Googlebot, Slurp (Yahoo), Bingbot, and any other spider that understands sitemaps should only visit it again if it's been updated in some manner.

That doesn't always happen, but the dumber spiders do get hooked on sessions so they won't necessarily get counted a bajillion times.

Offline Caeli

Re: Question about (Internet) spiders
« Reply #5 on: February 02, 2012, 04:13:57 PM »
How interesting.

You know, I've also seen spiders "posting" to a thread (or at least, that's what they're listed as doing). Obviously, they don't post, so why exactly does that message show up?

Online Vekseid

Re: Question about (Internet) spiders
« Reply #6 on: February 02, 2012, 04:21:43 PM »
The action description you see is triggered by how SMF understands the URL of whomever is visiting it.

Open up another browser where you aren't logged into Elliquiy. Copy the post link to a thread where you are logged in, paste it into the unlogged-in browser.

Now go back to your logged-in browser, go to 'who's online', and view guests. You're non-logged-in self is 'posting'.

Offline Caeli

Re: Question about (Internet) spiders
« Reply #7 on: February 02, 2012, 04:25:55 PM »
Oh, how fascinating!

These spiders, they intrigue me. >_>

Offline Gabby girl

Re: Question about (Internet) spiders
« Reply #8 on: February 10, 2012, 12:43:31 PM »
wait, what's a spider, and are they a bad thing here?   They don't bite, do they?  hehe    j/k on that last part :p

Offline OniyaTopic starter

  • StoreHouse of Useless Trivia
  • Oracle
  • Carnite
  • *
  • Join Date: Sep 2008
  • Location: Just bouncing through. Hi! City of Roses, Pennsylvania
  • Gender: Female
  • One bad Motokifuka. Also cute and FLUFFY!
  • My Role Play Preferences
  • View My Rolls
  • Referrals: 3
Re: Question about (Internet) spiders
« Reply #9 on: February 10, 2012, 12:55:08 PM »
Spiders (in the Internet sense) are little programs that search engines use to compile web pages for their search results.  As spiders help to steer people towards pages that match what they're looking for, spiders are actually rather beneficial.  However, as anything in the public area of Elliquiy can be 'seen' by spiders (and therefore, can be searched for), they are yet another reason to keep private stuff (like contact info) out of the public area.

Offline Beguile's Mistress

  • Time flies like an arrow ~ Fruit flies like a banana ~ Elliquiy's Fair-E Godmother
  • Dame
  • Carnite
  • *
  • Join Date: Jul 2009
  • Location: Faeleacanvald ~ The Steeler Nation ~ Home of Lord Stanley's Cup 2016 ~ She won't stay throwed! ~ 48\22-5\1\11-5\7
  • Gender: Female
  • Perpetual Notion Machine ~ 'What if...?'
  • My Role Play Preferences
  • View My Rolls
  • Referrals: 3
Re: Question about (Internet) spiders
« Reply #10 on: February 10, 2012, 01:28:29 PM »
I was pointed to this article a while back by one of my friends when I asked her about web crawlers.