Canīt save queue while analyzing?!?!?

Michael Riis Nielsen

10-05-2004 20:53:09

Hi, I've tried to save the queue for several times
It pauses all the threads except 2.

Its the same 2 threads that keeps comming back. evil

Can't I delete em? or should i restart the program and start over? so I can save again?

Best regards
Michael

Russ

10-05-2004 21:03:46

try to decrease the total number of threads. it seems that they hang up waiting for wininet.dll to allow connection. let me know if it worked for you.

Michael Riis Nielsen

10-05-2004 21:13:12

I've tried set both threads to 1, then only the Fetcher runs, and the crawler says it's idle

Michael

Russ

10-05-2004 21:34:12

does links queue counter show that there are still links in the queue. crawlers are supposed to be idle when there are no page links left to follow.

also, sometimes threads do not update their state when they quit, thus leaving their latest status message in the threads list. you may right-click on the threads list and select "Reset" to redraw the list. It might be the cause that makes you think that these threads do no stop when you press the pause button.

in general, when you press the pause button, you may save the queue once ther "disk" button is highlighted. and it is being highlighted when all threads are paused and it's okay to save the queue.

hope that helps,
Russ.

Michael Riis Nielsen

10-05-2004 21:44:08

Been there - done that! sorry

Right now I have 256612 links to follow... theres enough I think D
And when i Pause the threads the disk above is not highlightet.

I have a DSL connection, and connect with 15 fetchers and 15 crawlers.
They are all running and I cannot see the 2 files except when i pauses the threads - they show up again! and say they are analyzing the page.

I have also tried to reset the threads, but they still keep comming back!

No highlightet disk at all.

Could that be an idea to implement in the program to skip the file or somthing like that?

I have earlier tried to have so many links without that kind of problem.

Hope you can help...

Best regards
Michael

Russ

10-05-2004 22:02:23

256612 urls in the media queue, or in the links queue or is it combined number of links?

Michael Riis Nielsen

10-05-2004 22:03:59

Links Queue!

Michael Riis Nielsen

10-05-2004 22:05:14

I have 370791 in the media queue

Russ

10-05-2004 22:21:03

does it always show the same link when you press the pause button?
if this is the same link, can you e-mail me this link and I will try to look at its source code.

Michael Riis Nielsen

10-05-2004 22:34:29

sure
it is the same 2 links it stops at.

Michael Riis Nielsen

10-05-2004 22:36:04

sorry..
didnt see that you wantet emailed! email sent

Michael Riis Nielsen

11-05-2004 01:04:30

I have paused all the threads for aprox 1 hour now, and the save button is not highlightet yet.

Michael