Krdytkyu

I've created a script using python in association with scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a csv file other than using the built in command provided by scrapy cause when I do like this scrapy crawl torrentdata -o outputfile.csv -t csv I get a blank line in every alternate row in the csv file.

However, I thought to go in a slightly different way to achieve the same. Now, I get a data laden csv file in the right format when I run the following script. Most importantly I made use of with statement while creating a csv file so that when the writing is done the file gets automatically closed. I used crawlerprocess to execute the script from within an IDE.

My question: ain't it a better idea, if I comply with the way I tried below?

This is the working script:

import scrapy

from scrapy.crawler import CrawlerProcess

import csv



class TorrentSpider(scrapy.Spider):

    name = "torrentdata"

    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list

    itemlist = 



    def parse(self, response):

        for record in response.css('.browse-movie-bottom'):

            items = {}

            items["Name"] = record.css('.browse-movie-title::text').extract_first(default='')

            items["Year"] = record.css('.browse-movie-year::text').extract_first(default='')

            self.itemlist.append(items)



        with open("outputfile.csv","w", newline="") as f:

            writer = csv.DictWriter(f,['Name','Year'])

            writer.writeheader()

            for data in self.itemlist:

                writer.writerow(data)



c = CrawlerProcess({

    'USER_AGENT': 'Mozilla/5.0',   

})

c.crawl(TorrentSpider)

c.start() 

By putting the CSV exporting logic into the spider itself, you are re-inventing the wheel and not using all the advantages of Scrapy and its components and, also, making the crawling slower as you are writing to disk in the crawling stage every time the callback is triggered.

As you mentioned, the CSV exporter is built-in, you just need to yield/return items from the parse() callback:

import scrapy





class TorrentSpider(scrapy.Spider):

    name = "torrentdata"

    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list



    def parse(self, response):

        for record in response.css('.browse-movie-bottom'):

            yield {

                "Name": record.css('.browse-movie-title::text').extract_first(default=''),

                "Year": record.css('.browse-movie-year::text').extract_first(default='')

            }

Then, by running:

scrapy runspider spider.py -o outputfile.csv -t csv

(or the crawl command)

you would have the following in the outputfile.csv:

Name,Year

"Faith, Love & Chocolate",2018

Bennett's Song,2018

...

Tender Mercies,1983

You Might Be the Killer,2018