Temporary forum search engine

Onkel Neal · 05-02-06, 06:57 PM

Drebbel suggested using Google to search the forums. Look for the new search bar at top left.

blizzard_beast · 05-02-06, 08:36 PM

Great!

Will a more advanced search feature be implemented in the future? (That is, like the one we had before)

Gizzmoe · 05-02-06, 11:14 PM

I hope this is just a temporary solution...

Onkel Neal · 05-02-06, 11:25 PM

Yeah, as soon as Bill Gates calls me, we will have a dual core XEON server just for the forums

Gizzmoe · 05-02-06, 11:57 PM

A real answer would be nice!

Google is, as you know, absolutely no substitute for a real forum search engine.

First you said that phpbb sucks for big forums (search table problem), now you say that itÂ´s the current server hardware that canÂ´t handle the forum search? Well, if thatÂ´s really the case then let us know what it would cost to buy or rent a new server or to upgrade the old one, then we can can start donating money...

Onkel Neal · 05-03-06, 12:03 AM

Lol, I'm working on it, I hope to narrow down the answer. I have paid a "professional" server management company $300 last month, and so far they have not been able to fix a darn thing. Is it the server config? Is the load too great for the server? Who knows?? But the load now is nothing compared to the load when DW and SH3 came out. That's why I find it hard to believe there is so much load that the server cannot handle it

Last year at times we had 150 people on the forums at one time, now 50 is a lot.... yeah, tricky problem, I'm working on it.

Drebbel · 05-03-06, 04:34 AM

Quote:

Originally Posted by Gizzmoe

A real answer would be nice!

The real answer would be for someone to step forward to help out. That someone should really know about webservers, because at the moment it is unknown if the server problem is hardware or software related.

And we all know the solution starts with finding the problem. So if webservers have no (or atleast not much) secrets for you then please step forward and help subsim survive this storm :-)

Drebbel

Wildcat · 05-03-06, 08:47 AM

Your database size is probably too large. I can see literally hundreds of pages in the older forums. All the old stuff needs to be archived and put into seperate archive forums and the new stuff kept to a reasonable size of less than 20 pages. Anything over 20 pages old should be moved to the archives.

Drebbel · 05-03-06, 09:08 AM

Quote:

Originally Posted by Wildcat

Your database size is probably too large. I can see literally hundreds of pages in the older forums. All the old stuff needs to be archived and put into seperate archive forums and the new stuff kept to a reasonable size of less than 20 pages. Anything over 20 pages old should be moved to the archives.

Yep, Neal is going to do that in the weekend. Too bad that is neccesary.

Ppl walking on the moon and having a decent forum is still impossile :-)

Gizzmoe · 05-03-06, 09:55 AM

Quote:

Originally Posted by Wildcat

Anything over 20 pages old should be moved to the archives.

Just speculating here, but if the database size and/or the searchtable size is the problem, wouldnÂ´t it then be better to move much less pages to the archives? The SH3 forum for example has 380 pages, the archive needs to be searchable, now if you move 360 pages to the archive that would most likely cause trouble in archive database, right? Moving 50% posts of the SH3 posts instead of more than 90% sounds like a better idea in my opinion. Same goes for the Mods forum, 150 pages, move 75 pages instead of 130. There would be a better load-sharing between the two databases then.

Gizzmoe · 05-03-06, 09:58 AM

Quote:

Originally Posted by Gizzmoe

Quote:

Originally Posted by Wildcat

Anything over 20 pages old should be moved to the archives.

Just speculating here, but if the database size and/or the searchtable size is the problem, wouldnÂ´t it then be better to move much less pages to the archives? The SH3 forum for example has 380 pages, the archive needs to be searchable, now if you move 360 pages to the archive that would most likely cause trouble in archive database, right? Moving 50% of the SH3 posts instead of more than 90% sounds like a better idea in my opinion. Same goes for the Mods forum, 150 pages, move 75 pages instead of 130. There would be a better load-sharing between the two databases then.

Gizzmoe · 05-03-06, 10:00 AM

Grah!!! I want my Edit and Delete button back!!!

Jesper · 05-03-06, 10:48 AM

without knowing the actual system specs (hardware and OS) and how much traffic there is on the server (assuming its not only hosting www.subsim.com), and not knowing the actual symptoms other than the host is unreachable, does the system shutdown/crash, or just hangs while completing jobs ? or what is it that there is happening behind the curtain?

I'd say that it is not a database issue as I have seen forums with much higher post and member counts that doesnt seem to have these problems that subsim has.

If it was due to excessive load on mysql then moving topics from one forum to another forum using the same mysql isnt going to help one single bit, cause you only moves the load from 1 db to another on the same system, go figure guys

What that might help is to trim the topics and users, for instance prune all topics older than X days or with 0/1 reply, and prune all members with 3 or less posts.
Ban Search Engines by IP address/range, googlebot alone can stall any server for very long periods, especially if the forum it is indexing logs all visitor moves.
Force people to use small signatures, there are still a ton of people using half a page+ to show their signature which is outrageous and abuse of subsims goodwill to allow huge signatures (bytewise not sizewise, come on people think, or Neal will cut down the byte limit)...

Neal PM me server details and the httpd.conf file, the actual filesize of the mysql database(s) and ill take a look at it, also if you have a stats page, i'd like to see the daily bandwidth usage for 3-4 weeks or so

Gizzmoe · 05-03-06, 11:01 AM

Quote:

Originally Posted by Jesper

If it was due to excessive load on mysql then moving topics from one forum to another forum using the same mysql isnt going to help one single bit, cause you only moves the load from 1 db to another on the same system, go figure guys

I thought a second MySQL database woukd start a second instance of mysqld!?!

Quote:

Force people to use small signatures, there are still a ton of people using half a page+ to show their signature which is outrageous and abuse of subsims goodwill to allow huge signatures (bytewise not sizewise, come on people think, or Neal will cut down the byte limit)...

The signature images are not stored on Subsim.com, so thatÂ´s no problem.

Wildcat · 05-03-06, 12:58 PM

Well the problem arises when people make a search of a forum and it has to go through 3+ years of old posts to complete the search. If the old posts are split up into archives, they'll need to be searched seperately and it'll make it a lot easier on the server to do that.

05-02-06, 06:57 PM	#1
Onkel Neal Born to Run Silent Join Date: Jan 1997 Location: Cougar Trap, Texas Posts: 21,289 Downloads: 534 Uploads: 224	Temporary forum search engine Drebbel suggested using Google to search the forums. Look for the new search bar at top left.

05-03-06, 10:48 AM	#13
Jesper Torpedoman Join Date: Sep 2002 Location: Denmark (GMT+01.00) Posts: 112 Downloads: 0 Uploads: 0	without knowing the actual system specs (hardware and OS) and how much traffic there is on the server (assuming its not only hosting www.subsim.com), and not knowing the actual symptoms other than the host is unreachable, does the system shutdown/crash, or just hangs while completing jobs ? or what is it that there is happening behind the curtain? I'd say that it is not a database issue as I have seen forums with much higher post and member counts that doesnt seem to have these problems that subsim has. If it was due to excessive load on mysql then moving topics from one forum to another forum using the same mysql isnt going to help one single bit, cause you only moves the load from 1 db to another on the same system, go figure guys What that might help is to trim the topics and users, for instance prune all topics older than X days or with 0/1 reply, and prune all members with 3 or less posts. Ban Search Engines by IP address/range, googlebot alone can stall any server for very long periods, especially if the forum it is indexing logs all visitor moves. Force people to use small signatures, there are still a ton of people using half a page+ to show their signature which is outrageous and abuse of subsims goodwill to allow huge signatures (bytewise not sizewise, come on people think, or Neal will cut down the byte limit)... Neal PM me server details and the httpd.conf file, the actual filesize of the mysql database(s) and ill take a look at it, also if you have a stats page, i'd like to see the daily bandwidth usage for 3-4 weeks or so __________________ Jesper [JiD]: jesper@tidesofwar.net My Firefox & Thunderbird Your Firefox & Thunderbird

05-02-06, 08:36 PM	#2
blizzard_beast Watch Join Date: Apr 2006 Location: UK Posts: 20 Downloads: 0 Uploads: 0	Great! Will a more advanced search feature be implemented in the future? (That is, like the one we had before)

05-02-06, 11:14 PM	#3
Gizzmoe Silent Hunter Join Date: Mar 2005 Location: Germany Posts: 3,668 Downloads: 4 Uploads: 0	I hope this is just a temporary solution...

05-02-06, 11:25 PM	#4
Onkel Neal Born to Run Silent Join Date: Jan 1997 Location: Cougar Trap, Texas Posts: 21,289 Downloads: 534 Uploads: 224	Yeah, as soon as Bill Gates calls me, we will have a dual core XEON server just for the forums

05-02-06, 11:57 PM	#5
Gizzmoe Silent Hunter Join Date: Mar 2005 Location: Germany Posts: 3,668 Downloads: 4 Uploads: 0	A real answer would be nice! Google is, as you know, absolutely no substitute for a real forum search engine. First you said that phpbb sucks for big forums (search table problem), now you say that itÂ´s the current server hardware that canÂ´t handle the forum search? Well, if thatÂ´s really the case then let us know what it would cost to buy or rent a new server or to upgrade the old one, then we can can start donating money...

05-03-06, 12:03 AM	#6
Onkel Neal Born to Run Silent Join Date: Jan 1997 Location: Cougar Trap, Texas Posts: 21,289 Downloads: 534 Uploads: 224	Lol, I'm working on it, I hope to narrow down the answer. I have paid a "professional" server management company $300 last month, and so far they have not been able to fix a darn thing. Is it the server config? Is the load too great for the server? Who knows?? But the load now is nothing compared to the load when DW and SH3 came out. That's why I find it hard to believe there is so much load that the server cannot handle it Last year at times we had 150 people on the forums at one time, now 50 is a lot.... yeah, tricky problem, I'm working on it.

05-03-06, 08:47 AM	#8
Wildcat Engineer Join Date: Jan 2002 Posts: 215 Downloads: 69 Uploads: 0	Your database size is probably too large. I can see literally hundreds of pages in the older forums. All the old stuff needs to be archived and put into seperate archive forums and the new stuff kept to a reasonable size of less than 20 pages. Anything over 20 pages old should be moved to the archives.

05-03-06, 10:00 AM	#12
Gizzmoe Silent Hunter Join Date: Mar 2005 Location: Germany Posts: 3,668 Downloads: 4 Uploads: 0	Grah!!! I want my Edit and Delete button back!!!

05-03-06, 12:58 PM	#15
Wildcat Engineer Join Date: Jan 2002 Posts: 215 Downloads: 69 Uploads: 0	Well the problem arises when people make a search of a forum and it has to go through 3+ years of old posts to complete the search. If the old posts are split up into archives, they'll need to be searched seperately and it'll make it a lot easier on the server to do that.