{"id":1030,"date":"2015-03-11T19:00:03","date_gmt":"2015-03-11T19:00:03","guid":{"rendered":"https:\/\/nickv.codes\/?p=1030"},"modified":"2016-07-22T02:23:12","modified_gmt":"2016-07-22T02:23:12","slug":"scrapy-uploading-image-files-to-amazon-s3","status":"publish","type":"post","link":"https:\/\/nickv.codes\/blog\/scrapy-uploading-image-files-to-amazon-s3\/","title":{"rendered":"Scrapy &#8211; Uploading image files to Amazon S3"},"content":{"rendered":"<p>Scrapy has a nice built in feature to automatically download and store images that it comes across while scraping a website. There&#8217;s some <a href=\"http:\/\/doc.scrapy.org\/en\/latest\/topics\/images.html\" target=\"_blank\">great documentation<\/a> on how to get started. Theres also some undocumented code that allows you to store your images on Amazon S3. Here&#8217;s how to do that:<\/p>\n<p>in your <code>settings.py<\/code> file:<\/p>\n<pre class=\"prettyprint lang-python\">\nITEM_PIPELINES = {\n    'scrapy.contrib.pipeline.images.ImagesPipeline': 1\n}\n\n# This is going to be the amazon s3 bucket. \n# You need to use the below format so Scrapy \n# can parse it. !!Important don't forget to add \n# the trailing slash.\nIMAGES_STORE = 's3:\/\/my-bucket-name\/'\n\nIMAGES_EXPIRES = 180 # The amount of days until we re-download the image\n\nIMAGES_THUMBS = {\n    'small': (50, 50), # You can add as many of these as you want\n    'big': (300, 300)\n}\n\nAWS_ACCESS_KEY_ID = 'your-access-key'\nAWS_SECRET_ACCESS_KEY= 'your-secret-access-key'\n<\/pre>\n<p>For the sake of security I suggest creating a new user in the Amazon AWS interface and give that user only read\/write privileges to your bucket.<\/p>\n<p>Now we need to install a few packages that didn&#8217;t come by default with Scrapy:<\/p>\n<pre class=\"prettyprint lang-bsh\">\npip install pillow\npip intall boto\n<\/pre>\n<p>Pillow handles the image manipulation and boto will provide the library that connects to S3.<\/p>\n<p>Scrapy uses the <code>image_urls<\/code> key in your item to look for images it should download. This should be a list of image urls. Once downloaded Scrapy writes the details of the image location to the <code>images<\/code> key.<\/p>\n<p>Don&#8217;t forget to add these to your <code>items.py<\/code> file:<\/p>\n<pre class=\"prettyprint lang-python\">\nclass MyItem(scrapy.Item):\n    image_urls = scrapy.Field()\n    images = scrapy.Field()\n<\/pre>\n<p>Now don&#8217;t forget to actually populate the <code>image_urls<\/code> key during your crawl. Once you crawl your site final output will look something like this for a given item:<\/p>\n<pre class=\"prettyprint lang-json\">\n'image_urls': [u'http:\/\/example.com\/images\/tshirt.jpg'],\n'images': [{ 'checksum': '264d3bbdffd4ab3dcb8f234c51329da8',\n         'path': 'full\/069f409fd4cdb02248d726a625fecd8299e6055e.jpg',\n         'url': 'http:\/\/example.com\/images\/tshirt.jpg'}],\n<\/pre>\n<p>Now head on over to you amazon S3 bucket and have a look. Your images and thumbnails are all there!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scrapy has a nice built in feature to automatically download and store images that it comes across while scraping a website. There&#8217;s some great documentation on how to get started. Theres also some undocumented code that allows you to store your images on Amazon S3. Here&#8217;s how to do that: in your settings.py file: ITEM_PIPELINES [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1037,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[120,121],"tags":[124,123,125,122],"_links":{"self":[{"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/posts\/1030"}],"collection":[{"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/comments?post=1030"}],"version-history":[{"count":0,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/posts\/1030\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/media\/1037"}],"wp:attachment":[{"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/media?parent=1030"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/categories?post=1030"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nickv.codes\/wp-json\/wp\/v2\/tags?post=1030"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}