Java開發項目之手把手教你爬取並下載酷狗TOP500歌曲

作者：後山悟道人
來源：https://my.oschina.net/gllfeixiang/blog/2995570
 

是這樣的，之前買車送的垃圾記錄儀不能用了，這兩天狠心買了好點的記錄儀，帶導航、音樂、藍牙、4G等功能，尋思，既然有這些功能就利用起來，用4G聽歌有點奢侈，就準備去酷狗下點歌聽，居然都是需要辦會員才能下載，而且vip一月只能下載300首，我這麼窮又這麼摳怎麼可能衝會員，於是百度搜了下怎麼免費下載，都是python爬取，雖然也會一點，但是電腦上沒安裝python，再安裝再研究感覺有點費勁，於是就花了半小時做了這個爬蟲，技術一般，只記錄分析實現過程，大牛請繞行。其中用到了一些庫，包括：jsoup、HttpClient、net.sf.json大家可以自行去下載jar包。

1、分析是否能獲得TOP500歌單

首先，打開酷狗首頁查看酷狗TOP500，說好的500首，怎麼就只有22首呢？

是真的只讓看這些還是能找到其餘的呢，於是我就看了下這TOP500的鏈接：

https://www.kugou.com/yy/rank/home/1-8888.html?from=rank

可以看的出home後邊有個1，難道這是代表第一頁的意思？於是我就把1改成2，進入，果然進入了第二頁， 至此可以知道我們可以在網頁裡獲取這500首的歌單。

2、分析找到真正的mp3下載地址（這個有點繞）

點一個歌曲進入播放頁面，使用谷歌瀏覽器的控制檯的Elements，搜一下mp3，很輕鬆就定位到了MP3的位置。

但是使用java訪問的時候爬取的html裡卻沒有該mp3的文件地址，那麼這肯定是在該頁面的位置使用了js來加載mp3，那麼刷新下網頁，看網頁加載了哪些東西，加載的東西有點多，著重看一下js、php的請求，主要是看裡面有沒有mp3的地址，分析細節就不用說了。

最終我在列表的

https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&hash=667939C6E784265D541DEEE65AE4F2F8&album_id=0&_=1546235744251

這個請求裡發現了mp3的完整地址：

"play_url": "http:\/\/fs.w.kugou.com\/201812311325\/dcf5b6449160903c6ee48035e11434bb\/G128\/M08\/02\/09\/IIcBAFrZqf2ANOadADn94ubOmaU995.mp3",

那這個js是怎麼判斷是哪首歌的呢，那麼只可能是hash這個參數來決定歌曲的，然後到播放頁面裡找到這個hash的位置，是在下面的js裡：

var dataFromSmarty = [{"hash":"667939C6E784265D541DEEE65AE4F2F8","timelength":"237051","audio_name":"\\u767d\\u5c0f\\u767d - \\u6700\\u7f8e\\u5a5a\\u793c","author_name":"\\u767d\\u5c0f\\u767d","song_name":"\\u6700\\u7f8e\\u5a5a\\u793c","album_id":0}],//當前頁面歌曲信息
 playType = "search_single";//當前播放
 

在去java爬取該網頁，查看能否爬到這個hash，果然，爬取的html裡有這段js，到現在mp3的地址也找到了，歌單也找到了，那麼下一步就用程序實現就可以了。

3、java實現爬取酷狗mp3

先看一下爬取結果：

找到了資源，程序實現就好說了，其中使用到了自己寫的幾個工具類，自己整理點自己的工具類還是有好處的，以後遇到什麼問題就沒必要重新寫了，直接拿來用就可以了。沒什麼好說的了，下面直接貼出源碼。

SpiderKugou.java

package com.bing.spider;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.download.FileDownload;
import com.bing.html.HtmlManage;
import com.bing.http.HttpGetConnect;
import net.sf.json.JSONObject;
public class SpiderKugou {
 public static String filePath = "F:/music/";
 public static String mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&"
 + "hash=HASH&album_id=0&_=TIME";
 
 public static String LINK = "https://www.kugou.com/yy/rank/home/PAGE-8888.html?from=rank";
 //"https://www.kugou.com/yy/rank/home/PAGE-23784.html?from=rank";
 
 
 public static void main(String[] args) throws IOException {
 
 for(int i = 1 ; i < 23 ; i++){
 String url = LINK.replace("PAGE", i + "");
 getTitle(url);
 //download("https://www.kugou.com/song/mfy6je5.html");
 }
 }
 
 public static String getTitle(String url) throws IOException{
 HttpGetConnect connect = new HttpGetConnect();
 String content = connect.connect(url, "utf-8");
 HtmlManage html = new HtmlManage();
 Document doc = html.manage(content);
 Element ele = doc.getElementsByClass("pc_temp_songlist").get(0);
 Elements eles = ele.getElementsByTag("li");
 for(int i = 0 ; i < eles.size() ; i++){
 Element item = eles.get(i);
 String title = item.attr("title").trim();
 String link = item.getElementsByTag("a").first().attr("href");
 
 download(link,title);
 }
 return null;
 }
 
 public static String download(String url,String name) throws IOException{
 String hash = "";
 HttpGetConnect connect = new HttpGetConnect();
 String content = connect.connect(url, "utf-8"); 

 HtmlManage html = new HtmlManage();
 
 String regEx = ""hash":"[0-9A-Z]+"";
 // 編譯正則表達式
 Pattern pattern = Pattern.compile(regEx);
 Matcher matcher = pattern.matcher(content);
 if (matcher.find()) {
 hash = matcher.group();
 hash = hash.replace(""hash":"", "");
 hash = hash.replace(""", "");
 }
 
 String item = mp3.replace("HASH", hash);
 item = item.replace("TIME", System.currentTimeMillis() + "");
 System.out.println(item);
 String mp = connect.connect(item, "utf-8");
 
 mp = mp.substring(mp.indexOf("(") + 1, mp.length() - 3);
 
 JSONObject json = JSONObject.fromObject(mp);
 String playUrl = json.getJSONObject("data").getString("play_url");
 
 System.out.print(playUrl + " == ");
 FileDownload down = new FileDownload();
 down.download(playUrl, filePath + name + ".mp3");
 
 System.out.println(name + "下載完成");
 return playUrl;
 }
}

HttpGetConnect.java

package com.bing.http;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager; 

import javax.net.ssl.X509TrustManager;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.params.HttpParams; 
/**
* @說明：
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HttpGetConnect {
 
 /**
 * 獲取html內容
 * @param url
 * @param charsetName UTF-8、GB2312
 * @return
 * @throws IOException
 */
 public static String connect(String url,String charsetName) throws IOException{
 BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager();
 
 CloseableHttpClient httpclient = HttpClients.custom()
 .setConnectionManager(connManager)
 .build();
 String content = "";
 
 try{
 HttpGet httpget = new HttpGet(url);
 
 RequestConfig requestConfig = RequestConfig.custom()
 .setSocketTimeout(5000)
 .setConnectTimeout(50000)
 .setConnectionRequestTimeout(50000) 

 .build();
 httpget.setConfig(requestConfig);
 httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
 httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch");
 httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
 httpget.setHeader("Connection", "keep-alive");
 httpget.setHeader("Upgrade-Insecure-Requests", "1");
 httpget.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
 //httpget.setHeader("Hosts", "www.oschina.net");
 httpget.setHeader("cache-control", "max-age=0"); 
 
 CloseableHttpResponse response = httpclient.execute(httpget);
 
 int status = response.getStatusLine().getStatusCode();
 if (status >= 200 && status < 300) {
 
 HttpEntity entity = response.getEntity();
 InputStream instream = entity.getContent();
 BufferedReader br = new BufferedReader(new InputStreamReader(instream,charsetName));
 StringBuffer sbf = new StringBuffer();
 String line = null;
 while ((line = br.readLine()) != null){
 sbf.append(line + "\n");
 }
 br.close();
 content = sbf.toString();
 } else {
 content = "";
 }
 
 }catch(Exception e){
 e.printStackTrace();
 }finally{
 httpclient.close();
 }
 //log.info("content is " + content);
 return content;
 }
 private static Log log = LogFactory.getLog(HttpGetConnect.class);
}

HtmlManage.java

package com.bing.html; 

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.http.HttpGetConnect;
/**
* @說明：
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HtmlManage {
 
 public Document manage(String html){
 Document doc = Jsoup.parse(html);
 return doc;
 }
 
 public Document manageDirect(String url) throws IOException{
 Document doc = Jsoup.connect( url ).get();
 return doc;
 }
 
 public List manageHtmlTag(Document doc,String tag ){
 List list = new ArrayList();
 
 Elements elements = doc.getElementsByTag(tag);
 for(int i = 0; i < elements.size() ; i++){
 String str = elements.get(i).html();
 list.add(str);
 }
 return list;
 }
 
 public List manageHtmlClass(Document doc,String clas ){
 List list = new ArrayList();
 
 Elements elements = doc.getElementsByClass(clas);
 for(int i = 0; i < elements.size() ; i++){ 

 String str = elements.get(i).html();
 list.add(str);
 }
 return list;
 }
 
 public List manageHtmlKey(Document doc,String key,String value ){
 List list = new ArrayList();
 
 Elements elements = doc.getElementsByAttributeValue(key, value);
 for(int i = 0; i < elements.size() ; i++){
 String str = elements.get(i).html();
 list.add(str);
 }
 return list;
 }
 
 private static Log log = LogFactory.getLog(HtmlManage.class);
}

FileDownload.java

package com.bing.download;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet; 

import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
/**
* @說明：
* @author: gaoll
* @CreateTime:2014-11-20
* @ModifyTime:2014-11-20
*/
public class FileDownload {
 
 /**
 * 文件下載
 * @param url 鏈接地址
 * @param path 要保存的路徑及文件名
 * @return
 */
 public static boolean download(String url,String path){
 
 boolean flag = false;
 
 CloseableHttpClient httpclient = HttpClients.createDefault();
 RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000)
 .setConnectTimeout(2000).build();
 HttpGet get = new HttpGet(url);
 get.setConfig(requestConfig);
 
 BufferedInputStream in = null;
 BufferedOutputStream out = null;
 try{
 for(int i=0;i<3;i++){
 CloseableHttpResponse result = httpclient.execute(get);
 System.out.println(result.getStatusLine());
 if(result.getStatusLine().getStatusCode() == 200){
 in = new BufferedInputStream(result.getEntity().getContent());
 File file = new File(path);
 out = new BufferedOutputStream(new FileOutputStream(file));
 byte[] buffer = new byte[1024];
 int len = -1;
 while((len = in.read(buffer,0,1024)) > -1){
 out.write(buffer,0,len);
 }
 flag = true;
 break;
 }else if(result.getStatusLine().getStatusCode() == 500){
 continue ;
 }
 }
  

 }catch(Exception e){
 e.printStackTrace();
 flag = false;
 }finally{
 get.releaseConnection();
 try{
 if(in != null){
 in.close();
 }
 if(out != null){
 out.close();
 }
 }catch(Exception e){
 e.printStackTrace();
 flag = false;
 }
 }
 return flag;
 }
 private static Log log = LogFactory.getLog(FileDownload.class);
}

到這就結束了，有可能有些代碼沒貼全，主要代碼已經差不多，應該可以跑起來，多多指教。

閱讀更多 Java面經 的文章

關鍵字: 下載 會員 酷狗音樂