准备好 Ubuntu14.04 System.
1.安装 scrapy:
1) 按照这个:
Ubuntu packages
New in version 0.10.
Scrapinghub(http://scrapinghub.com/) publishes apt-gettable packages which are generally fresher than those in Ubuntu, and more stable too since they’re continuously built from Github repo(https://github.com/scrapy/scrapy) (master & stable branches) and so they contain the latest bug fixes.
To use the packages:
1.Import the GPG key used to sign Scrapy packages into APT keyring:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7
2.Create /etc/apt/sources.list.d/scrapy.list file using the following command:
echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt/sources.list.d/scrapy.list
3.Update package lists and install the scrapy package:
sudo apt-get update && sudo apt-get install scrapy
Note
Repeat step 3 if you are trying to upgrade Scrapy.
Warning
python-scrapy is a different package provided by official debian repositories, it’s very outdated and it isn’t supported by Scrapy team.
但是可能会遇到下面的问题:
尝试了以下方式解决了该问题:
1>>sudo apt-get install python-pip【如果已经安装pip就不必操作此步】
2>>sudo pip install scrapy
2)另外,还有可能遇到pyasn1版本的问题:
解决方法:
1>>sudo apt-get install python-pip
2>>sudo pip install pyasn1 --upgrade
2.安装 redis-server
1)按照 http://www.linuxdiyf.com/linux/16857.html
但是默认server绑定了本地IP,不支持远程访问,需要Redis服务器的配置文件redis.conf。
1)修改redis.conf:
sudo vi /etc/redis/redis.conf
2)注释bind:
#bind 127.0.0.1
3)修改后,重启Redis服务器:
sudo /etc/init.d/redis-server restart
【注意】:千万不要在windows下面修改之后再copy覆盖Ubuntu中的redis.conf,之前因为用SecureCRT,它对vi的支持不太好,所以用rz和sz命令把ubuntu中的文件下载到windows本地进行修改之后再上传覆盖,由于删掉了原来的文件,修改后的文件由于编码问题导致各种报错,最后redis崩溃了。尝试了各种方法均不好使,最后卸载了redis,由于存在某些文件的残留,导致redis安装失败。最后只能重装系统,重新开始。
经验教训:得到一个教训在做重要操作之前一定要备份,然后即使犯了错误,也可以返回到最后一次能正确运行的地方。
使用Docker部署Scrapy爬虫:http://www.linuxdiyf.com/linux/14696.html
Ubuntu 14.04 64bit上安装Scrapy:http://www.linuxdiyf.com/linux/13980.html
Ubuntu14.04下安装scrapy:http://www.linuxdiyf.com/linux/13816.html
如何在Ubuntu 14.04 LTS安装网络爬虫工具:Scrapy:http://www.linuxdiyf.com/linux/9985.html
CentOS 6.4安装搭建Scrapy 0.22环境:http://www.linuxdiyf.com/linux/13205.html