引言

在編程中經(jīng)常會遇到圖片等數(shù)據(jù)集將圖片等數(shù)據(jù)以URL形式存儲在txt文檔中，為便于后續(xù)的分析，需要將其下載下來，并按照文件夾分類存儲。本文以Github中Alexander Kim提供的圖片分類數(shù)據(jù)集為例，下載其提供的圖片樣本并分類保存

Python 3.6.5，Anaconda， VSCode

1. 下載數(shù)據(jù)集文件

建立項目文件夾，下載上述Github項目中的raw_data文件夾，并保存至項目目錄中。

2. 獲取樣本文件位置

編寫get_doc_path.py，根據(jù)根目錄位置，獲取目錄及其子目錄所有數(shù)據(jù)集文件

import osdef get_file(root_path, all_files={}): ’’’ 遞歸函數(shù)，遍歷該文檔目錄和子目錄下的所有文件，獲取其path ’’’ files = os.listdir(root_path) for file in files: if not os.path.isdir(root_path + ’/’ + file): # not a dir all_files[file] = root_path + ’/’ + file else: # is a dir get_file((root_path+’/’+file), all_files) return all_filesif __name__ == ’__main__’: path = ’./raw_data’ print(get_file(path))

3. 下載文件

3.1 讀取url列表并

for filename, path in paths.items(): print(’reading file: {}’.format(filename)) with open(path, ’r’) as f: lines = f.readlines() url_list = [] for line in lines:url_list.append(line.strip(’n’)) print(url_list)

3.2 創(chuàng)建文件夾

foldername = './picture_get_by_url/pic_download/{}'.format(filename.split(’.’)[0])if not os.path.exists(folder_path): print('Selected folder not exist, try to create it.') os.makedirs(folder_path)

3.3 下載圖片

def get_pic_by_url(folder_path, lists): if not os.path.exists(folder_path): print('Selected folder not exist, try to create it.') os.makedirs(folder_path) for url in lists: print('Try downloading file: {}'.format(url)) filename = url.split(’/’)[-1] filepath = folder_path + ’/’ + filename if os.path.exists(filepath): print('File have already exist. skip') else: try:urllib.request.urlretrieve(url, filename=filepath) except Exception as e:print('Error occurred when downloading file, error message:')print(e)

4. 完整源碼

4.1 get_doc_path.py

import osdef get_file(root_path, all_files={}): ’’’ 遞歸函數(shù)，遍歷該文檔目錄和子目錄下的所有文件，獲取其path ’’’ files = os.listdir(root_path) for file in files: if not os.path.isdir(root_path + ’/’ + file): # not a dir all_files[file] = root_path + ’/’ + file else: # is a dir get_file((root_path+’/’+file), all_files) return all_filesif __name__ == ’__main__’: path = ’./raw_data’ print(get_file(path))

4.2 get_pic.py

import get_doc_pathimport osimport urllib.requestdef get_pic_by_url(folder_path, lists): if not os.path.exists(folder_path): print('Selected folder not exist, try to create it.') os.makedirs(folder_path) for url in lists: print('Try downloading file: {}'.format(url)) filename = url.split(’/’)[-1] filepath = folder_path + ’/’ + filename if os.path.exists(filepath): print('File have already exist. skip') else: try:urllib.request.urlretrieve(url, filename=filepath) except Exception as e:print('Error occurred when downloading file, error message:')print(e)if __name__ == '__main__': root_path = ’./picture_get_by_url/raw_data’ paths = get_doc_path.get_file(root_path) print(paths) for filename, path in paths.items(): print(’reading file: {}’.format(filename)) with open(path, ’r’) as f: lines = f.readlines() url_list = [] for line in lines:url_list.append(line.strip(’n’)) foldername = './picture_get_by_url/pic_download/{}'.format(filename.split(’.’)[0]) get_pic_by_url(foldername, url_list)

4.3 運(yùn)行結(jié)果

執(zhí)行g(shù)et_pic.py當(dāng)程序意外停止或再次執(zhí)行時，程序會自動跳過文件夾中已下載的文件，繼續(xù)下載未下載的內(nèi)容

{‘urls_drawings.txt’: ‘./picture_get_by_url/raw_data/drawings/urls_drawings.txt’, ‘urls_hentai.txt’: ‘./picture_get_by_url/raw_data/hentai/urls_hentai.txt’, ‘urls_neutral.txt’: ‘./picture_get_by_url/raw_data/neutral/urls_neutral.txt’, ‘urls_porn.txt’: ‘./picture_get_by_url/raw_data/porn/urls_porn.txt’, ‘urls_sexy.txt’: ‘./picture_get_by_url/raw_data/sexy/urls_sexy.txt’}reading file: urls_drawings.txtTry downloading file: http://41.media.tumblr.com/xxxxxx.jpgTry downloading file: http://41.media.tumblr.com/xxxxxx.jpgTry downloading file: http://ak1.polyvoreimg.com/cgi/img-thing/size/l/tid/xxxxxx.jpgError occurred when downloading file, error message:HTTP Error 502: No data received from server or forwarderTry downloading file: http://akicocotte.weblike.jp/gaugau/xxxxxx.jpgTry downloading file: http://animewriter.files.wordpress.com/2009/01/nagisa-xxxxxx-xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpgTry downloading file: http://cdn.awwni.me/xxxxxx.jpg

后注：由于樣本數(shù)據(jù)集內(nèi)容的問題，上述地址以xxxxx代替具體地址，案例項目也已經(jīng)失效，但是方法仍然可以借鑒

20.9.23更新：數(shù)據(jù)集地址：https://github.com/ZQ-Qi/nsfw_data_scrapper，單純?yōu)榱藢W(xué)習(xí)和實(shí)踐本文代碼的可以下載該數(shù)據(jù)集進(jìn)行嘗試

到此這篇關(guān)于Python根據(jù)URL地址下載文件并保存至對應(yīng)目錄的實(shí)現(xiàn)的文章就介紹到這了,更多相關(guān)Python URL下載文件內(nèi)容請搜索好吧啦網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持好吧啦網(wǎng)！