1、工具

2、具體方法

1、使用python2.7編寫爬取腳本

這裡除了正常的爬取操作，還增加了獨立的參數設定。如果沒有參數，爬取的數據就在當前目錄下；如果有參數，可以設定保存目錄、保存文件名後綴。這樣的話，這個腳本既可以單獨使用，也可以配合sh定時任務使用。

雙色球爬取代碼grab500_ssq.py內容：

# -*- coding:utf-8 -*-
import re
import urllib
import time
import sys
 
datapath = sys.path[0]
datasuffix = 'txt'
if (len(sys.argv)>1):
 datapath = sys.argv[1]
 datasuffix = sys.argv[2]
 
def getHtml(url):
 html = urllib.urlopen(url)
 return html.read()
 
 
html = getHtml("http://zx.500.com/ssq/")
 
 
reg = ['

([0-9]\d*).*

']
reg.append('

([0-9]\d*)

')
reg.append('

([0-9]\d*)

')

outstr = "";
for i in range(len(reg)):
page = re.compile(reg[i])
rs = re.findall(page,html)
for j in range(len(rs)):
outstr+= rs[j] + ","

#print time.strftime('%Y-%m-%d',time.localtime(time.time()))+":"+outstr[:-1]

with open(datapath+'/lot_500_ssq.'+datasuffix, 'a') as f:
f.write(time.strftime('%Y-%m-%d',time.localtime(time.time()))+":"+outstr[:-1]+'\n')

大樂透爬取代碼grab500_dlt.py內容：

# -*- coding:utf-8 -*-
import re
import urllib
import time
import sys
datapath = sys.path[0]
datasuffix ='txt'
if(len(sys.argv)>1):
 datapath = sys.argv[1]
 datasuffix = sys.argv[2]
defgetHtml(url):
 html = urllib.urlopen(url)
return html.read()
html = getHtml("http://zx.500.com/dlt/")
reg =['

([0-9]\d*).*

']
reg.append('

([0-9]\d*)

')
reg.append('

([0-9]\d*)

')
outstr ="";
for i in range(len(reg)):
page = re.compile(reg[i])
rs = re.findall(page,html)
for j in range(len(rs)):
outstr+= rs[j]+","

#print time.strftime('%Y-%m-%d',time.localtime(time.time()))+":"+outstr[:-1]
with open(datapath+'/lot_500_dlt.'+datasuffix,'a')as f:
f.write(time.strftime('%Y-%m-%d',time.localtime(time.time()))+":"+outstr[:-1]+'\n')

2、編寫一個執行的sh腳本

我們需要編寫執行python的sh腳本bwb_lottery_everyday.sh，要注意的是sh的date獲取的星期天值是0而不是7，而crontab則可以設定0或者7。

#!/bin/sh
basepath=$(cd `dirname $0`; pwd) #shell's dir
datapath=$basepath'/lotterydata' #shell's datadir
datasuffix='txt' #datasuffix
 
a=`date -d "${date}" +%w`
if [ $a -eq 1 ] || [ $a -eq 3 ] || [ $a -eq 6 ]; then
 python "${basepath}/grab500_ssq.py" $datapath $datasuffix
elif [ $a -eq 2 ] || [ $a -eq 4 ] || [ $a -eq 0 ]; then
 python "${basepath}/grab500_dlt.py" $datapath $datasuffix
fi

3、編寫一個主sh腳本

編寫一個主要的sh腳本bwb_lottery_main.sh，執行清理和設定的工作。需要注意的是，這裡直接使用了系統的/etc/crontab文件來達到週期執行的目的，其實並不太好，但crontab -e的方法很難自動化，所以只能設定為系統任務。

#!/bin/sh
cronfile="/etc/crontab" #debian cronfile
basepath=$(cd `dirname $0`; pwd) #shell's dir
datapath=$basepath'/lotterydata' #shell's datadir
datasuffix='txt' #datasuffix
crontaskname="bwb_lottery_everyday.sh" #shell's name
crontasktime="0 23\t* * 1-4,6-7" #crontab task run time,default everyday except friday 23:00
  

echo "checking..."
if [ ! -f ${cronfile} ]; then 
 echo "crontab file $cronfile doesn't exsits.\nplease check file or modify shell setting and run shell again." 
 exit 1
fi
 
pyver=`python -V 2>&1|awk '{print $2}'|awk -F '.' '{print $1}'`
if [ $pyver != '2' ]; then
 echo "python2(.7) is needed."
 exit 1
fi
 
 
echo "writing crontab file..."
if [ `grep -c "${crontaskname}" ${cronfile}` -eq '0' ]; then
 echo "${crontasktime}\troot\t${basepath}/${crontaskname}">>${cronfile}
else
 sed -i "s#^.*${crontaskname}.*#${crontasktime}\troot\t${basepath}/${crontaskname}#" ${cronfile}
fi
/etc/init.d/cron restart
 
echo "making data dir..."
if [ ! -d "${datapath}" ]; then
 mkdir ${datapath}
else
 if [ ! -d "${datapath}/bak" ]; then
 mkdir "${datapath}/bak"
 else
 mv ${datapath}/*.${datasuffix} ${datapath}/bak/ 2>/dev/null
 fi
fi
 
echo "changing permission..."
chmod +x "$basepath/$crontaskname"
chmod +w -R $datapath
 
echo "finished!"