Go back to previous page
Forum URL: http://www.dombom.com/cgi-bin/dcforum/dcboard.cgi
Forum Name: The New MadBomber Marketing and SEO Forum
Topic ID: 581
Message ID: 42
#42, RE: Creating a Datafeed....
Posted by Top Hat Bob on Jan-23-08 at 02:43 PM
In response to message #40
>How did you do that?
>. . .
>
>3. The challenge is that their site uses a database so you
>have question marks in the URL. This is hated by httrack.

I use Visual Web Spider to crawl a site. I haven't had a problem with databases. Sometimes I may have to crawl a site twice. Once to get the urls I need, then load those urls and crawl them directly.

It will find a needle in a hay stack. By that I mean if your affiliate site is about dogs and you want a feed about collars, it can retrieve only pages that have collar mentioned on them.

Once the pages are stored on your hard drive, it's just a matter of pulling the lines of text you want for feeds.

As I mentioned to Faraz, there are a lot of ways to clean text. I simply rename my text file to csv, open it with Excel and use the ASAP plugin to clean the data. Look at the "Clean Web Imported Data" and the "Advanced character removal" options. There are easier and better ways but this is what I use.

Ernie's solution posted above should do the same thing - albeit easier since he is offering to do it for you.