java 爬虫下载酷狗top500
标签：query code tostring rank www download from nload 其他
想下载歌曲，可app，网站啥的都需要会员，作为一个穷屌丝，没钱啊。所以想搞个代码去下载歌曲，
打开酷狗top500：http://www.kugou.com/yy/rank/home/1-8888.html
看到只有22个，有看了url猜测分页果然：把url改为：http://www.kugou.com/yy/rank/home/2-8888.html； 就进入了第二页
然后点击歌曲播放页面：http://www.kugou.com/song/#hash=3337C18539D5BB00D8027D653D536A35&album_id=32440418
使用谷歌浏览器的控制台的Elements搜索MP3
然而使用java请求，结果没有找到MP3，于是想到看看是不是其他请求或js加载的，然后查看entwork，找MP3
最终找到：https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery19106214312060791911_1573866413695&hash=3337C18539D5BB00D8027D653D536A35&album_id=32440418&dfid=2AMaYT1qKRQF0US0Ii20bXyy&mid=98350860ab960276d13883f7d07d364d&platid=4&_=1573866413696
然后查看参数：
使用postman测试查看必要参数为：
callback=jQuery191027067069941080546_1546235744250
hash=_HASH_
album_id=_ALBUM_ID_
mid=98350860ab960276d13883f7d07d364d
刷新另外一首歌，看到callback和mid没有变，所以只要找到hash和album_id就可以了
查看hash和album_id和播放页面url的相同，于是数据应该是从top500页面传过来的
使用java访问top500的页面发下一段字符串：
[{"Hash":"7707BE115CF9131E3AEF782D294155D4","FileName":"u963fu60a0u60a0 - u4e00u751fu4e0eu4f60u64e6u80a9u800cu8fc7","timeLen":239.464,"privilege":10,"size":3832017,"album_id":32922861,"encrypt_id":"11cb4g06"},{"Hash":"3337C18539D5BB00D8027D653D536A35","FileName":"u963fu5197 - u4f60u7684u7b54u6848","timeLen":219,"privilege":10,"size":3519376,"album_id":32440418,"encrypt_id":"113a6795"},{"Hash":"FAEDD01C425118BA343648B5AF35861F","FileName":"en - u56a3u5f20","timeLen":254.04,"privilege":10,"size":4065240,"album_id":26433357,"encrypt_id":"ybj0x09"},{"Hash":"33EB8FE0DC9F70D9F7FE4CB77305D5A8","FileName":"u6d77u6765u963fu6728u3001u963fu5477u62c9u53e4u3001u66f2u6bd4u963fu4e14 - u522bu77e5u5df1","timeLen":280.111,"privilege":10,"size":4482365,"album_id":16324799,"encrypt_id":"uajki71"},{"Hash":"1952AF6E49AF16B4C130282B8A40EEE7","FileName":"u845bu4e1cu742a - u60acu6eba","timeLen":197.12,"privilege":10,"size":3154481,"album_id":23773597,"encrypt_id":"10fdd762"},{"Hash":"E908C26B69C2873C75CFCEABB5B81F2B","FileName":"u6768u5c0fu58ee - u6211u627fu8ba4u6211u81eau5351","timeLen":269.348,"privilege":0,"size":4310039,"album_id":30562954,"encrypt_id":"10a05t84"},{"Hash":"ED8DBD8AE97359912A8AEC71C61758D2","FileName":"u6768u5c0fu58ee - u4e00u4e2au4ebau633au597d","timeLen":270.027,"privilege":10,"size":4320906,"album_id":26827357,"encrypt_id":"ygc0db4"},{"Hash":"B30DF1A53446B717C35629952E9B7156","FileName":"u6768u987au9ad8u3001u51efu5c0fu6674 - u4e0du7231u6211u5c31u522bu4f24u5bb3u6211","timeLen":253.44,"privilege":10,"size":4055502,"album_id":27400705,"encrypt_id":"ypwfo0a"},{"Hash":"247ED2BAA3AE6D3DBA48801F802459F3","FileName":"u4e8eu5609u4e50 - u9003u7231","timeLen":242.018,"privilege":10,"size":3875491,"album_id":13882049,"encrypt_id":"sulfsb0"},{"Hash":"B221F5355343714DC6AA7647B62F9B38","FileName":"u5468u6770u4f26 - u65adu4e86u7684u5f26","timeLen":297,"privilege":10,"size":4761507,"album_id":973367,"encrypt_id":"70mo63"},{"Hash":"B6880A36D02561912C53831ACF305D0D","FileName":"u5b9du77f3Gemu3001u9648u4f1fu9706 - u91ceu72fcDisco","timeLen":240.039,"privilege":10,"size":3841212,"album_id":30181741,"encrypt_id":"102ilgdd"},{"Hash":"04E4C1D0AFB9DEEF3B0834AA1F71B654","FileName":"u5d14u4f1fu7acb - u9152u9189u7684u8774u8776","timeLen":206.184,"privilege":10,"size":3299669,"album_id":26382011,"encrypt_id":"yavyv32"},{"Hash":"BE7C2E688ED155FA6A6B45392FD0FA50","FileName":"Tones And I - Dance Monkey","timeLen":209.815,"privilege":10,"size":3357510,"album_id":26441038,"encrypt_id":"xcw85d8"},{"Hash":"EBBDCAA92236F5877FAD2A91C91564F9","FileName":"u5f20u9753u9896 - u5e7bu7eb1u4e4bu7075","timeLen":248.032,"privilege":0,"size":3968984,"album_id":33385138,"encrypt_id":"11i24me5"},{"Hash":"DB66B6258041852A7B856729FF3A4893","FileName":"u738bu4f73u6768 - u9057u61be","timeLen":255.033,"privilege":10,"size":4081121,"album_id":27347801,"encrypt_id":"yp9nc64"},{"Hash":"517E05B8121A9E7D6E5629B41CCA61C4","FileName":"u97f3u9619u8bd7u542cu3001u8d75u65b9u5a67 - u8292u79cd","timeLen":216.032,"privilege":10,"size":3456984,"album_id":22135843,"encrypt_id":"x20jh9b"},{"Hash":"D46AC87092F83685353E564B1F5C9BD3","FileName":"u963fu60a0u60a0 - u5ff5u65e7","timeLen":229.877,"privilege":10,"size":3678626,"album_id":27580208,"encrypt_id":"ys2nmca"},{"Hash":"5784F2CBA32B65D5E48E4C415F843342","FileName":"u9ec4u9704u96f2 - u5de6u624bu6307u6708 (u7eafu4eabu7248)","timeLen":185.086,"privilege":10,"size":2974138,"album_id":12245979,"encrypt_id":"svuie28"},{"Hash":"07A55D0F8FF088BDCA36B836F7FE115F","FileName":"u674eu6615u878du3001u6a0au6850u821fu3001u674eu51efu7a20 - u4f60u7b11u8d77u6765u771fu597du770b","timeLen":172.068,"privilege":10,"size":2753687,"album_id":19428374,"encrypt_id":"w7pn29f"},{"Hash":"A017B769F3D653AC063AE20639C0020F","FileName":"u8521u5065u96c5 - u7ea2u8272u9ad8u8ddfu978b","timeLen":206.68,"privilege":10,"size":3321862,"album_id":978459,"encrypt_id":"39nred"},{"Hash":"6CB8F530E00F6CFEF4A3371FD3F0BEB8","FileName":"u5c0fu963fu67ab - u6700u8fdcu7684u4f60u662fu6211u6700u8fd1u7684u7231","timeLen":136.15,"privilege":10,"size":2178865,"album_id":25406316,"encrypt_id":"xzj9add"},{"Hash":"BBDCDDCBD58FF85B60E728726A4B3C9A","FileName":"u9708u4e39(u6d6au54e5) - u9003u7231","timeLen":242.233,"privilege":0,"size":3876263,"album_id":32104019,"encrypt_id":"10vik0d0"}]
里边存有hash和album_id，然后就是字符串获取了转换了 ，当然也发现了一个好东西：歌曲名称：FileName
然后就是代码了：
package com.kg; import net.sf.json.JSONArray; import net.sf.json.JSONObject; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; import java.io.IOException; import java.util.ArrayList; import java.util.Date; import java.util.List; import java.util.regex.Matcher; import java.util.regex.Pattern; /** * @ClassName: SpiderKugou.java * @Description: TODO * @author 小猪聪聪 * @version V1.0 * @Date 2019年11月15日 下午5:00:33 */ public class SpiderKugou { public static String filePath = "D:/music/"; public static String mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&" "hash=_HASH_&album_id=_ALBUM_ID_&mid=98350860ab960276d13883f7d07d364d"; public static String LINK = "http://www.kugou.com/song/#hash=_HASH_&album_id=_ALBUM_ID_"; // "https://www.kugou.com/yy/rank/home/PAGE-23784.html?from=rank"; public static void main(String[] args) throws IOException { int index = 4; d("http://www.kugou.com/yy/rank/home/" index "-8888.html"); } public static String d(String url) throws IOException { HttpGetConnect connect = new HttpGetConnect(); String content = connect.connect(url, "utf-8"); String hashStr = content.substring(content.indexOf("[{"), content.indexOf("}]") 2); JSONArray hashArr = JSONArray.fromObject(hashStr); System.out.println(hashArr); for (int i = 0; i < hashArr.size(); i ) { JSONObject item = JSONObject.fromObject(hashArr.get(i)); String hash = item.get("Hash").toString(); String albumId = item.get("album_id").toString(); String fileName = item.get("FileName").toString(); String itemUrl = mp3.replace("_HASH_", hash).replace("_ALBUM_ID_", albumId); download(itemUrl, fileName,i); } return null; } public static String download(String url, String name,int i) throws IOException { HttpGetConnect connect = new HttpGetConnect(); String mp = connect.connect(url, "utf-8"); mp = mp.substring(mp.indexOf("(") 1, mp.length() - 2); JSONObject json = JSONObject.fromObject(mp); String playUrl = json.getJSONObject("data").getString("play_url"); String song_name = json.getJSONObject("data").getString("song_name"); FileDownload down = new FileDownload(); down.download(playUrl, filePath song_name ".mp3"); System.out.println(name " ---- >>> 下载完成"); return playUrl; } }
package com.kg; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.http.HttpEntity; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.impl.conn.BasicHttpClientConnectionManager; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; /** * @ClassName: HttpGetConnect.java * @Description: TODO * @author 小猪聪聪 * @version V1.0 * @Date 2019年11月15日 下午5:01:28 */ public class HttpGetConnect { /** * 获取html内容 * * @param url * @param charsetName UTF-8、GB2312 * @return * @throws IOException */ public static String connect(String url, String charsetName) throws IOException { BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager(); CloseableHttpClient httpclient = HttpClients.custom().setConnectionManager(connManager).build(); String content = ""; try { HttpGet httpget = new HttpGet(url); RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(5000).setConnectTimeout(50000) .setConnectionRequestTimeout(50000).build(); httpget.setConfig(requestConfig); httpget.setHeader("Accept", "text/html,application/xhtml xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch"); httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8"); httpget.setHeader("Connection", "keep-alive"); httpget.setHeader("Upgrade-Insecure-Requests", "1"); httpget.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"); // httpget.setHeader("Hosts", "www.oschina.net"); httpget.setHeader("cache-control", "max-age=0"); CloseableHttpResponse response = httpclient.execute(httpget); int status = response.getStatusLine().getStatusCode(); if (status >= 200 && status < 300) { HttpEntity entity = response.getEntity(); InputStream instream = entity.getContent(); BufferedReader br = new BufferedReader(new InputStreamReader(instream, charsetName)); StringBuffer sbf = new StringBuffer(); String line = null; while ((line = br.readLine()) != null) { sbf.append(line ""); } br.close(); content = sbf.toString(); } else { content = ""; } } catch (Exception e) { e.printStackTrace(); } finally { httpclient.close(); } // log.info("content is " content); return content; } private static Log log = LogFactory.getLog(HttpGetConnect.class); }
package com.kg; import java.io.*; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; /** * @ClassName: FileDownload.java * @Description: TODO * @author 小猪聪聪 * @version V1.0 * @Date 2019年11月15日 下午5:02:22 */ public class FileDownload { /** * 文件下载 * * @param url 链接地址 * @param path 要保存的路径及文件名 * @return */ public static boolean download(String url, String path) throws IOException { new WriteText().writeToText(url); new WriteText().writeToText(path); boolean flag = false; CloseableHttpClient httpclient = HttpClients.createDefault(); RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000).setConnectTimeout(2000).build(); HttpGet get = new HttpGet(url); get.setConfig(requestConfig); BufferedInputStream in = null; BufferedOutputStream out = null; try { for (int i = 0; i < 3; i ) { CloseableHttpResponse result = httpclient.execute(get); System.out.println(result.getStatusLine()); if (result.getStatusLine().getStatusCode() == 200) { in = new BufferedInputStream(result.getEntity().getContent()); File file = new File(path); out = new BufferedOutputStream(new FileOutputStream(file)); byte[] buffer = new byte[1024]; int len = -1; while ((len = in.read(buffer, 0, 1024)) > -1) { out.write(buffer, 0, len); } flag = true; break; } else if (result.getStatusLine().getStatusCode() == 500) { continue; } } } catch (Exception e) { e.printStackTrace(); flag = false; } finally { get.releaseConnection(); try { if (in != null) { in.close(); } if (out != null) { out.close(); } } catch (Exception e) { e.printStackTrace(); flag = false; } } return flag; } private static Log log = LogFactory.getLog(FileDownload.class); }
package com.kg; import java.io.*; /** * @ClassName: aa.java * @Description: TODO * @author 小猪聪聪 * @version V1.0 * @Date 2019年11月15日 下午5:02:46 */ public class WriteText { public void writeToText(String musicInfo) throws IOException { String path = "D:\music\music_info\music_info.txt"; File file = new File(path); if (!file.exists()) { file.getParentFile().mkdirs(); } file.createNewFile(); // write FileWriter fw = new FileWriter(file, true); BufferedWriter bw = new BufferedWriter(fw); //写入到txt并自动换行 bw.write(musicInfo "rn"); bw.flush(); bw.close(); fw.close(); // read FileReader fr = new FileReader(file); BufferedReader br = new BufferedReader(fr); String str = br.readLine(); } }
package com.kg; import java.io.IOException; /** * @ClassName: Test.java * @Description: TODO * @author 小猪聪聪 * @version V1.0 * @Date 2019年11月15日 下午5:03:13 */ public class Test { public static void main(String[] args) throws IOException { System.out.println("aa"); new WriteText().writeToText("12376rn"); } }
问题：
下载中发现：只能下载73首歌曲，然后慢慢想问题，看到 https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&hash=7A5C31C00EB66499DE0F72BB67097111&album_id=19615336&mid=98350860ab960276d13883f7d07d364d
的返回值有问题，然后找问题，发现没有play_url返回了，于是用postman测试一样，继续找，找不到，下载不了了，问题。。
此时无意中刷新了一个页面，发现了一个东西，验证，没错，需要验证，还是图片验证，此时想到，可能服务器会检测同一台机器的访问次数，到一定次数了就需要验证，
于是把代码改为：
int index = 3; d("http://www.kugou.com/yy/rank/home/" index "-8888.html");
遇到的问题：
一次只加载一页，当遇到异常时，在刷新下页面，继续。。。
如果想一次全部加载：
方式1：
写代码验证,但他是图片验证，不好写，暂时放下，
方式2：
此时想到服务器是用那种方式计数的，ip，mid，或者其他的。。。。然后这个mid干嘛用的
在发现：下载的歌曲有些与名称不对，于是想到，歌曲名称获取不对，看到获取下载路径中的数据有歌曲名称，使用这个试试：
欢迎各位评论，留言
java 爬虫下载酷狗top500
标签：query code tostring rank www download from nload 其他
温馨提示: 本文由杰米博客推荐，转载请保留链接: https://www.jmwww.net/file/biancheng/12085.html
