Search:   Namespaces Discussions .NET v1.1 Feedback

Other ways to scrape a webpage Messages   Related Types This message was discovered on ASPFriends.com 'ngfx-io' list. Responses highlighted in red are from those people who are likely to be able to contribute good, authoratitive information to this discussion. They include Microsoft employees, MVP's and others who IMHO contribute well to these kinds of discussions. Devin Rader Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin Reply to this message...        dave wanta (VIP) Other ways to scrape a webpageSystem.Net.Sockets is the only other way I can think of... Create a new socket then Socket.Connect() should get to you the remote server. However, you will also get back all of the http headers, which HttpWebRequest handles nicely for you. --dave ----- Original Message ----- From: Devin Rader To: ngfx-io Sent: Thursday, April 25, 2002 10:01 AM Subject: [ngfx-io] Other ways to scrape a webpage Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT Reply to this message...        Peter Brunone Other ways to scrape a webpageDevin, What exactly are you trying to do, i.e. what is it about the HttpWebRequest method that doesn't work for you? This may be helpful in finding an appropriate solution. Regards, Peter -----Original Message----- From: Devin Rader [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 10:01 AM To: ngfx-io Subject: [ngfx-io] Other ways to scrape a webpage Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin Reply to this message...        Devin Rader Ya..I know..I wish I could use the WebResponse object, but there seems to be a problem when you try to use it in a threadpool. It seems to eat up all the available threads. I'll try the Socket way... devin -----Original Message----- From: dave wanta [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 10:20 AM To: ngfx-io Subject: [ngfx-io] Re: Other ways to scrape a webpage System.Net.Sockets is the only other way I can think of... Create a new socket then Socket.Connect() should get to you the remote server. However, you will also get back all of the http headers, which HttpWebRequest handles nicely for you. --dave ----- Original Message ----- From: Devin Rader <mailto:Click here to reveal e-mail address> To: ngfx-io <mailto:Click here to reveal e-mail address> Sent: Thursday, April 25, 2002 10:01 AM Subject: [ngfx-io] Other ways to scrape a webpage Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin | [ngfx-io] member Click here to reveal e-mail address <mailto:Click here to reveal e-mail address> = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp <http://www.aspfriends.com/aspfriends/ngfx-io.asp> = JOIN/QUIT | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT Reply to this message...        Devin Rader See my other reply to Dave about the ThreadPool problem. -----Original Message----- From: Peter Brunone [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 10:30 AM To: ngfx-io Subject: [ngfx-io] RE: Other ways to scrape a webpage Devin, What exactly are you trying to do, i.e. what is it about the HttpWebRequest method that doesn't work for you? This may be helpful in finding an appropriate solution. Regards, Peter -----Original Message----- From: Devin Rader [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 10:01 AM To: ngfx-io Subject: [ngfx-io] Other ways to scrape a webpage Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT Reply to this message...        Jeff Widmer Are you closing your response object?? Dim objResponse As HttpWebResponse 'Do the work here... objResponse.Close() I have had problems creating response and request objects in a loop. Closing and destroying the httpwebresponse and httpwebrequest objects in each loop fixed the problem. Not sure if this is relevant to a threadpool. -Jeff -----Original Message----- From: Devin Rader [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 8:38 AM To: ngfx-io Subject: [ngfx-io] Re: Other ways to scrape a webpage Ya..I know..I wish I could use the WebResponse object, but there seems to be a problem when you try to use it in a threadpool. It seems to eat up all the available threads. I'll try the Socket way... devin -----Original Message----- From: dave wanta [mailto:Click here to reveal e-mail address] Sent: Thursday, April 25, 2002 10:20 AM To: ngfx-io Subject: [ngfx-io] Re: Other ways to scrape a webpage System.Net.Sockets is the only other way I can think of... Create a new socket then Socket.Connect() should get to you the remote server. However, you will also get back all of the http headers, which HttpWebRequest handles nicely for you. --dave ----- Original Message ----- From: Devin Rader To: ngfx-io Sent: Thursday, April 25, 2002 10:01 AM Subject: [ngfx-io] Other ways to scrape a webpage Can anyone suggest other ways to scrape a webpage other that the HttpWebRequest/Response class? Thanks! devin | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT | [ngfx-io] member Click here to reveal e-mail address = YOUR ID | http://www.aspfriends.com/aspfriends/ngfx-io.asp = JOIN/QUIT Reply to this message...     System.Net.HttpWebRequest System.Net.HttpWebResponse System.Net.Sockets.Socket System.Net.WebResponse System.Threading.ThreadPool Ad MBR BootFX Best-of-breed application framework for .NET projects, developed by Matthew Baxter-Reynolds and MBR IT

Copyright © Matthew Baxter-Reynolds 2001-2008. '.NET 247 Software Development Services' is a trading style of MBR IT Solutions Ltd. Contact Us - Terms of Use - Privacy Policy - www.dotnet247.com