带你用python写一个下载文件的小工具

带你用python写一个下载文件的小工具

Python

关于Python,相信很多人都有所耳闻。经过近三十年的发展,再加上近几年AI的兴起,Python作为一种科学语言的流行程度急剧上升。而Python语法简单、语句清晰的特点,可以让程序员把精力集中在业务逻辑上,减轻我们的压力。毕竟人生苦短,我用Python。看看Python之父--Guido van Rossum的发量,在对比下Java之父——James Gosling。。。

带你用python写一个下载文件的小工具

Python之父

带你用python写一个下载文件的小工具

Java之父


开发环境

  • Python3.6.5
  • pip(Python包管理工具,Python3.4以上版本都会自带)
  • 一个顺手的编辑器(像这种一百多行代码的小工具,随便一个编辑器都可以,推荐vsCode,毕竟微软大法好)

需求分析

我们需要做一个下载文件并压缩保存的小工具,最好能多个目的地址同时下载。所以我们会用到以下功能:

  • ftp(引入paramiko包,Python支持ftp的包)
  • threading(Python对多线程支持的包)
  • 配置文件(需要引入json包,来解析json格式的配置文件,json格式简单明了而且解析方便,所以选择json格式配置文件)
  • os(要操作目录)
  • 需要记录日志(logging,Python日志相关的包)
  • zipfile(Python压缩zip文件需要用到的包)

开始撸代码

引入必要的包

<code>import paramiko
import threading
import json
import os
import sys from stat
import S_ISDIR as isdir
import logging
import zipfile
import time/<code>

获取ftp连接

<code>#建立连接,获取sftp句柄

def sftp_connect(username,password,host,port):

global logger

client = None

sftp = None

try:

client = paramiko.Transport((host,port))

except Exception as error:

logger.error(error)

else:

try:

client.connect(username=username, password=password)

except Exception as error:

logger.error(error)

else:

sftp = paramiko.SFTPClient.from_transport(client)

return client,sftp/<code>

断开ftp连接

<code>#断开连接

def disconnect(client):

global logger

try:

client.close()

except Exception as error:

logger.error(error)



def check_local(localPath):

global logger

if not os.path.exists(localPath):

try:

os.mkdir(localPath)

except IOError as err:

logger.error(err)/<code>

从ftp服务器下载文件

<code>#下载方法

def downLoad(client, sftp, remote, local):

global logger

#检查远程文件

try:

result = sftp.stat(remote)

except IOError as err:

error = '[ERROR %s] %s: %s' %(err.errno,os.path.basename(os.path.normpath(remote)),err.strerror)

logger.error(error)

else:

if isdir(result.st_mode):

dirname = os.path.basename(os.path.normpath(remote))

local = os.path.join(local, dirname)

#local = local.replace("\\\","/")


check_local(local)

for file in sftp.listdir(remote):

sub_remote = os.path.join(remote, file)

sub_remote = sub_remote.replace("\\\","/")

downLoad(client, sftp,sub_remote,local)

else:

if os.path.isdir(local):

local = os.path.join(local, os.path.basename(remote))

try:

sftp.get(remote, local)

except IOError as err:

logger.error(err)

else:

logger.info('[get] %s %s %s', remote, '==>', local)

lock.acquire()

global finish

finish += 1

lock.release()

logger.info('已下载 [%d] 个文件', finish)/<code>

压缩文件

<code>#压缩文件

def writeAllFileToZip(absDir,zipFile):

global logger

for f in os.listdir(absDir):

absFile=os.path.join(absDir,f) #子文件的绝对路径

if os.path.isdir(absFile): #判断是文件夹,继续深度读取。

zipFile.write(absFile) #在zip文件中创建文件夹

logger.info('写入 %s 到压缩包 %s 成功', absFile, zipFile.filename)

writeAllFileToZip(absFile,zipFile) #递归操作

else: #判断是普通文件,直接写到zip文件中。

zipFile.write(absFile)

logger.info('写入 %s 到压缩包 %s 成功', absFile, zipFile.filename)

return/<code>

主方法(程序入口)

<code>if __name__ == "__main__":

#日志

logging.basicConfig(level=logging.INFO,format ='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger()

logFormatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

handler = logging.FileHandler("log.txt")

handler.setLevel(logging.INFO)

handler.setFormatter(logFormatter)

console = logging.StreamHandler()

console.setLevel(logging.INFO)

console.setFormatter(logFormatter)

logger.addHandler(handler)


#logger.addHandler(console)

#锁

lock = threading.Lock()

#条件

finish = 0

#读取配置文件

configFile = open('config.json')

configs = json.loads(configFile.read())

configFile.close()



logger.info('工作目录:%s', configs['workDir'])

#检查本地工作目录(父目录)

check_local(configs['workDir'])

threadList = []

for config in configs['threads']:

logger.info("=======================配置信息 start=============================")

logger.info("HostAddress %s",config['HostAddress'])

logger.info('Port %s', config['Port'])

logger.info('Username %s',config['Username'])

logger.info('Password %s',config['Password'])

logger.info('RemotePath %s',config['RemotePath'])

logger.info('LocalPath %s',config['LocalPath'])

logger.info("=======================配置信息 end=============================")

client,sftp = sftp_connect(config['Username'],config['Password'],config['HostAddress'],config['Port'])

#创建本地目录

check_local(config['LocalPath'])

#多线程

t = threading.Thread(target=downLoad, args=(client, sftp, config['RemotePath'], config['LocalPath']))

t.start()

threadList.append(t)

#单线程

#downLoad(client, sftp, config['RemotePath'], config['LocalPath'])

for t in threadList:

t.join()



#压缩

# while threading.active_count() == 1:

logger.info('=======================准备开始压缩==========================')

#time.sleep(3)

logger.info('==========================开始压缩。。。==========================')

check_local(configs['zipDir'])

zipFilePath = os.path.join(configs['zipDir'], configs['zipName'] + '.zip')

zipFile = zipfile.ZipFile(zipFilePath, 'w', zipfile.ZIP_DEFLATED, allowZip64=True)

absZipFileDir = configs['workDir']

writeAllFileToZip(absZipFileDir, zipFile)

logger.info('==========================压缩完成==========================')

#保留终端


print("Press Enter to continue ...")

input()/<code>

配置文件

<code>{

"workDir": "F:/temp",

"zipDir": "F:/result",

"zipName": "rrrr",

"threads": [{

"HostAddress": "10.7.5.69",

"Port": 22,

"Username": "ftpuser",

"Password": "ftpuser",

"RemotePath": "/app/ftpuser/100_OrdMember_2019",

"LocalPath": "F:/temp/1"

},

{

"HostAddress": "10.7.5.69",

"Port": 22,

"Username": "ftpuser",

"Password": "ftpuser",

"RemotePath": "/app/ftpuser/100_OrdMember_2019",

"LocalPath": "F:/temp/2"

},

{

"HostAddress": "10.7.5.69",


"Port": 22,

"Username": "ftpuser",

"Password": "ftpuser",

"RemotePath": "/app/ftpuser/100_OrdMember_2019",

"LocalPath": "F:/temp/3"

}

]

}/<code>

源码讲解

这里我们做的其实是一个Python脚本,像是shell脚本。程序入口在主方法,进入主方法后我们干了几件事:

声明日志,固定了日志输出的格式、日志等级和输出日志的目的地(日志文件和终端)。输出日志到日志文件用于保存日志,便于日后查看,输出到终端是让用户及时了解程序在做什么,做到什么程度了。

声明一个锁。这个所是为了下载文件计数,保证下载文件时,只有一个线程可以操作变量finish。

读取配置文件。想要开始执行,得先获取咱们的配置文件,让程序知道它连接那一台服务器,下载哪个目录下的文件到本地哪个目录,等等……

检查本地工作目录。为了减少用户配置的错误,避免他/她配了一个不存在的地址导致程序找不到目录而引发错误,所以我们要判断目录是否存在,是否需要创建目录。

根据配置文件threads中的配置,循环创建线程,完成读取本线程需要的配置文件、连接ftp、下载文件的操作。

压缩文件。拼接文件目录,保证压缩文件的目录关系不变,调用压缩文件方法。

美化结束动作,如果没有这一步,程序跑完就会关闭当前终端,导致用户无法了解程序是跑完了,还是中间出了问题关闭了。输出一段文字告诉用户可以按Enter来结束程序,并且用input()监听用户的输入,达到保留终端的效果。

查看日志文件

<code>2020-01-19 13:35:33,253 - root - INFO - 工作目录: F:/temp
2020-01-19 13:35:33,254 - root - INFO - =======================配置信息 start=============================
2020-01-19 13:35:33,256 - root - INFO - HostAddress 10.7.5.69
2020-01-19 13:35:33,257 - root - INFO - Port 22
2020-01-19 13:35:33,258 - root - INFO - Username ftpuser
2020-01-19 13:35:33,259 - root - INFO - Password ftpuser
2020-01-19 13:35:33,260 - root - INFO - RemotePath /app/ftpuser/100_OrdMember_2019
2020-01-19 13:35:33,260 - root - INFO - LocalPath F:/temp/1
2020-01-19 13:35:33,261 - root - INFO - =======================配置信息 end=============================
2020-01-19 13:35:33,290 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_5.5)
2020-01-19 13:35:33,638 - paramiko.transport - INFO - Authentication (password) successful!
2020-01-19 13:35:33,656 - paramiko.transport.sftp - INFO - [chan 0] Opened sftp connection (server version 3)
2020-01-19 13:35:33,659 - root - INFO - =======================配置信息 start=============================

2020-01-19 13:35:33,660 - root - INFO - HostAddress 10.7.5.69
2020-01-19 13:35:33,661 - root - INFO - Port 22
2020-01-19 13:35:33,661 - root - INFO - Username ftpuser
2020-01-19 13:35:33,662 - root - INFO - Password ftpuser
2020-01-19 13:35:33,663 - root - INFO - RemotePath /app/ftpuser/100_OrdMember_2019
2020-01-19 13:35:33,664 - root - INFO - LocalPath F:/temp/2
2020-01-19 13:35:33,665 - root - INFO - =======================配置信息 end=============================
2020-01-19 13:35:33,680 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_5.5)
2020-01-19 13:35:33,987 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_2019222 ==> F:/temp/1\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222
2020-01-19 13:35:33,987 - root - INFO - 已下载 [1] 个文件
2020-01-19 13:35:34,011 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_201922 ==> F:/temp/1\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922
2020-01-19 13:35:34,012 - root - INFO - 已下载 [2] 个文件
2020-01-19 13:35:34,032 - paramiko.transport - INFO - Authentication (password) successful!
2020-01-19 13:35:34,047 - paramiko.transport.sftp - INFO - [chan 0] Opened sftp connection (server version 3)
2020-01-19 13:35:34,050 - root - INFO - =======================配置信息 start=============================
2020-01-19 13:35:34,051 - root - INFO - HostAddress 10.7.5.69
2020-01-19 13:35:34,052 - root - INFO - Port 22
2020-01-19 13:35:34,052 - root - INFO - Username ftpuser
2020-01-19 13:35:34,053 - root - INFO - Password ftpuser
2020-01-19 13:35:34,054 - root - INFO - RemotePath /app/ftpuser/100_OrdMember_2019
2020-01-19 13:35:34,055 - root - INFO - LocalPath F:/temp/3
2020-01-19 13:35:34,056 - root - INFO - =======================配置信息 end=============================
2020-01-19 13:35:34,060 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/tmp001.xls ==> F:/temp/1\\100_OrdMember_2019\\tmp001.xls
2020-01-19 13:35:34,071 - root - INFO - 已下载 [3] 个文件
2020-01-19 13:35:34,072 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_5.5)
2020-01-19 13:35:34,095 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019111 ==> F:/temp/1\\100_OrdMember_2019\\100_OrdMember_2019111
2020-01-19 13:35:34,096 - root - INFO - 已下载 [4] 个文件
2020-01-19 13:35:34,118 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_2019222 ==> F:/temp/2\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222
2020-01-19 13:35:34,118 - root - INFO - 已下载 [5] 个文件
2020-01-19 13:35:34,123 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_201911 ==> F:/temp/1\\100_OrdMember_2019\\100_OrdMember_201911
2020-01-19 13:35:34,124 - root - INFO - 已下载 [6] 个文件
2020-01-19 13:35:34,258 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_201922 ==> F:/temp/2\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922
2020-01-19 13:35:34,258 - root - INFO - 已下载 [7] 个文件
2020-01-19 13:35:34,449 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/tmp001.xls ==> F:/temp/2\\100_OrdMember_2019\\tmp001.xls
2020-01-19 13:35:34,450 - root - INFO - 已下载 [8] 个文件
2020-01-19 13:35:34,464 - paramiko.transport - INFO - Authentication (password) successful!
2020-01-19 13:35:34,479 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019111 ==> F:/temp/2\\100_OrdMember_2019\\100_OrdMember_2019111
2020-01-19 13:35:34,480 - root - INFO - 已下载 [9] 个文件
2020-01-19 13:35:34,480 - paramiko.transport.sftp - INFO - [chan 0] Opened sftp connection (server version 3)
2020-01-19 13:35:34,508 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_201911 ==> F:/temp/2\\100_OrdMember_2019\\100_OrdMember_201911
2020-01-19 13:35:34,509 - root - INFO - 已下载 [10] 个文件
2020-01-19 13:35:34,552 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_2019222 ==> F:/temp/3\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222
2020-01-19 13:35:34,552 - root - INFO - 已下载 [11] 个文件

2020-01-19 13:35:34,576 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019/100_OrdMember_201922 ==> F:/temp/3\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922
2020-01-19 13:35:34,576 - root - INFO - 已下载 [12] 个文件
2020-01-19 13:35:34,624 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/tmp001.xls ==> F:/temp/3\\100_OrdMember_2019\\tmp001.xls
2020-01-19 13:35:34,624 - root - INFO - 已下载 [13] 个文件
2020-01-19 13:35:34,649 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_2019111 ==> F:/temp/3\\100_OrdMember_2019\\100_OrdMember_2019111
2020-01-19 13:35:34,649 - root - INFO - 已下载 [14] 个文件
2020-01-19 13:35:34,675 - root - INFO - [get] /app/ftpuser/100_OrdMember_2019/100_OrdMember_201911 ==> F:/temp/3\\100_OrdMember_2019\\100_OrdMember_201911
2020-01-19 13:35:34,675 - root - INFO - 已下载 [15] 个文件
2020-01-19 13:35:34,677 - root - INFO - =======================准备开始压缩==========================
2020-01-19 13:35:34,678 - root - INFO - ==========================开始压缩。。。==========================
2020-01-19 13:35:34,682 - root - INFO - 写入 F:/temp\\1 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,684 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,686 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,696 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,698 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,700 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\100_OrdMember_201911 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,702 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\100_OrdMember_2019111 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,716 - root - INFO - 写入 F:/temp\\1\\100_OrdMember_2019\\tmp001.xls 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,717 - root - INFO - 写入 F:/temp\\2 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,718 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,720 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,723 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,738 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,740 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\100_OrdMember_201911 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,742 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\100_OrdMember_2019111 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,745 - root - INFO - 写入 F:/temp\\2\\100_OrdMember_2019\\tmp001.xls 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,768 - root - INFO - 写入 F:/temp\\3 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,770 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,772 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\100_OrdMember_2019 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,774 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_201922 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,776 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\100_OrdMember_2019\\100_OrdMember_2019222 到压缩包 F:/result\\rrrr.zip 成功

2020-01-19 13:35:34,790 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\100_OrdMember_201911 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,792 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\100_OrdMember_2019111 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,794 - root - INFO - 写入 F:/temp\\3\\100_OrdMember_2019\\tmp001.xls 到压缩包 F:/result\\rrrr.zip 成功
2020-01-19 13:35:34,794 - root - INFO - ==========================压缩完成==========================/<code>

使用pyinstaller打包


<code>pyinstaller -F -i .\\downRecordFile.ico recoderFileDownload.py/<code>

使用Python32位pyinstaller出来的是运行于32位Windows操作系统的程序,Python64位则适用于64位Windows操作系统。


分享到:


相關文章: