很多情况下,尤其是在公司工作,内部网络与外部网络是不相通的,因此经常需要在本地构建源。
1.APT本地源的搭建
本节介绍使用apt-mirror搭建apt本地源的方法,简单几步就行:
(注:请先准备100G以上的空间来保存镜像文件,是的你没有看错,100G,建议选个快些的网络吧...)
首先,安装apt-mirror
$ sudo apt-get install apt-mirror
apt-get命令的源配置位于/etc/apt/sources.list,而apt-mirror命令的源配置位于/etc/apt/mirror.list,很好记吧。
(mirror.list文件中配置的就是下载apt源所有文件的地址,建议采用国内的源,这里以阿里云的源为例,其他的见后边附1)
############# config ##################
# 以下注释的内容都是默认配置,如果需要自定义,取消注释修改即可
set base_path /var/spool/apt-mirror
#
# 镜像文件下载地址
# set mirror_path $base_path/mirror
# 临时索引下载文件目录,也就是存放软件仓库的dists目录下的文件(默认即可)
# set skel_path $base_path/skel
# 配置日志(默认即可)
# set var_path $base_path/var
# clean脚本位置
# set cleanscript $var_path/clean.sh
# 架构配置,i386/amd64,默认的话会下载跟本机相同的架构的源
set defaultarch amd64
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
# 下载线程数
set nthreads 20
set _tilde 0
#
############# end config ##############
# Ali yun(这里没有添加deb-src的源)
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
clean http://mirrors.aliyun.com/ubuntu
准备好之后就可以同步了:
$ sudo apt-mirror
然后就是漫长的等待了,同步的镜像文件目录为/var/spool/apt-mirror/mirror/mirrors.aliyun.com/ubuntu/,当然如果增加了其他的源,在/var/spool/apt-mirror/mirror目录下还有其他的地址为名的目录。
同步完成之后,放到httpd/apache2等http服务中就行,以apache2为例:
$ sudo apt-get install apache2
由于Apache2的默认网页文件目录位于/var/www/html,因此,可以做个软链接
$ sudo ln -s /var/spool/apt-mirror/mirror/mirrors.aliyun.com/ubuntu /var/www/html/ubuntu
然后就可以通过如下地址访问了
http://[host]:[port]/ubuntu
如果看到地址下边是dist/和pool/两个目录就OK了,然后放到/etc/apt/sources.list中试试吧
# Local Source
deb http://[host]:[port]/ubuntu/ trusty main restricted universe multiverse
deb http://[host]:[port]/ubuntu/ trusty-security main restricted universe multiverse
deb http://[host]:[port]/ubuntu/ trusty-updates main restricted universe multiverse
deb http://[host]:[port]/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://[host]:[port]/ubuntu/ trusty-backports main restricted universe multiverse
执行更新命令"apt-get update",然后享受风驰电掣的速度吧。
2.Pypi本地源的搭建
使用Bandersnatch进行完整同步
本节介绍使用Bandersnatch进行镜像同步的方法,还是善意提醒一下,您需要准备大约120G以上的空间,对,您又没看错,这还是2014年的情况。
假定您已经安装python。
首先安装virtualenvwrapper
$ sudo apt-get install virtualenvwrapper
然后建立一个virtualenv,并进入该env
$ sudo mkvirtualenv pypi-mirror
$ sudo workon pypi-mirror
安装bandersnatch
(pypi-mirror)$ pip install -r https://bitbucket.org/pypa/bandersnatch/raw/stable/requirements.txt
(pypi-mirror)$ pip install bandersnatch
(pypi-mirror)$ bandersnatch mirror
第一次运行“bandersnatch mirror”命令会提示您进行相关配置,编辑/etc/bandersnatch.conf,比如修改directory项
[mirror]
; The directory where the mirror data will be stored.
directory = /var/spool/pypi
......
......
然后再次运行“bandersnatch mirror”命令即可开始下载。
只用pip2pi工具仅同步所需的Pypi包
全部下载所有的镜像文件有时候确实是比较耗时的,那么也可以仅同步所需的Pypi包。
首先,安装pip2pi工具
$ pip install pip2pi
或
$ git clone https://github.com/wolever/pip2pi
$ cd pip2pi
$ python setup.py install
然后,创建存放软件包的仓库
$ sudo mkdir /var/spool/pypi-mirror
之后就可以下载软件包了,可以单个下载,也可以批量下载,比如
$ pip2tgz /var/spool/pypi-mirror/ routes==1.12.3
$ pip2tgz /var/spool/pypi-mirror/ -r requirements.txt
requirements.txt文件是一个待下载软件包列表,比如openstack需求的包列表(https://github.com/openstack/requirements/blob/master/global-requirements.txt)。
软件包下载后,建立索引:
$ dir2pi /var/spool/pypi-mirror/
若软件需求有更新,可以如下更新索引:
$ pip2acmeco uliweb=0.2.6
$ pip2acmeco -r list/requirements.txt
最后,将库地址配置给http服务就行了,方法同apt的类似,也可以采用自己的服务软件,比如nginx配置,这里介绍nginx的配置方法。
给Nginx服务器添加虚拟主机配置:
server {
listen 80;
server_name pypi.diyoms.com;
root /opt/python/soft/pypi.diyoms.com;
location /{
autoindex on;
autoindex_exact_size off; #显示文件的大小
autoindex_localtime on; #显示文件时间
#limit_rate_after 5m; #5分钟后下载速度限制为200k
limit_rate 200k;
}
access_log logs/pypi.diyoms.com.access.log main;
}
源的使用方法及国内其他源的地址见附1。
3.RubyGem本地源的搭建
本节介绍rubygems-mirror工具下载本地源的方法,再次善意提醒,gem源的大小也很惊人,80G,您是不是已经习惯了,哭晕在厕所好嘛~
首先安装rubygems-mirror:
$ sudu apt-get install rubygems-mirror -r
编辑~/.gem/.mirrorrc(最开始可能没有这个文件,创建一个就行)
---
- from: http://rubygems.org
to: /var/spool/rubygems
parallelism: 10
retries: 3
delete: false
skiperror: true
主要的参数有from和to,当然,from也可以换成国内源(见附1),会快一些;to的话是镜像文件保存的位置。
此时执行命令gem mirror即可进行镜像下载啦。
附1:国内源
1.APT源
修改/etc/apt/sources.list(建议先备份:sudo cp /etc/apt/sources.list /etc/apt/sources.list.back),选择添加如下内容,然后运行"apt-get update"即可(列表是针对ubuntu14.04的,即trusty版本):
# 阿里源(个人推荐):
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
# 163源:
deb http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse
# 搜狐源:
deb http://mirrors.sohu.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.sohu.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.sohu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.sohu.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.sohu.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.sohu.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.sohu.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.sohu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.sohu.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.sohu.com/ubuntu/ trusty-backports main restricted universe multiverse
# 官方源:
deb http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
另外,最近在学习CI过程中陆续添加的源,其中的内容相对于ubuntu的apt源版本会新很多:
# Puppet(puppetlab官方安装源)
deb http://apt.puppetlabs.com trusty main
# Jenkins(Jenkins安装方式的deb源)
deb http://pkg.jenkins-ci.org/debian binary/
# Oracle Java(可以安装Oraclejava7/8甚至9)
deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
# Git(目前apt的git版本是1.8.x,这是最新的git安装源)
deb http://ppa.launchpad.net/git-core/ppa/ubuntu trusty main
# GitLab
deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu trusty main
2.Pypi源
Pypi的国内镜像源有
http://pypi.douban.com/ 豆瓣
http://pypi.hustunique.com/ 华中理工大学
http://pypi.sdutlinux.org/ 山东理工大学
http://pypi.mirrors.ustc.edu.cn/ 中国科学技术大学
https://pypi.tuna.tsinghua.edu.cn/ 清华大学
对于pip这种在线安装的方式来说,很方便,但网络不稳定的话很要命。使用国内镜像相对好一些,
如果想手动指定源,可以在pip后面跟-i 来指定源,比如用豆瓣的源来安装web.py框架:
pip install web.py -i http://pypi.douban.com/simple
注意后面要有/simple目录!
要配置成默认的话,需要创建或修改配置文件(linux的文件在~/.pip/pip.conf,windows在%HOMEPATH%\pip\pip.ini),修改内容为:
[global] index-url = http://pypi.douban.com/simple
这样在使用pip来安装时,会默认调用该镜像。
3.RubyGem源
http://ruby.taobao.org 淘宝
使用方法(添加淘宝源,删除官方源):
$ gem sources -a http://ruby.taobao.org
$ gem sources -r https://rubygems.org/