Skip to content

Commit c4cdcb3

Browse files
YangzhediYangzhedi
committed
更新了股票爬虫，增加创建文件夹功能
1 parent 68fb746 commit c4cdcb3

9 files changed

Lines changed: 167 additions & 104 deletions

File tree

‎by-bs4/shares/601600中国铝业/Y2016S2.txt‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
2016-06-30 3.78 3.79 3.75 3.77 -0.01 -0.26 603,245 22,732 1.06 0.55
2+
2016-06-29 3.77 3.80 3.75 3.78 0.02 0.53 1,019,381 38,479 1.33 0.93
3+
2016-06-28 3.73 3.77 3.70 3.76 0.01 0.27 515,963 19,266 1.87 0.47
4+
2016-06-27 3.69 3.78 3.69 3.75 0.03 0.81 740,401 27,687 2.42 0.68
5+
2016-06-24 3.79 3.80 3.66 3.72 -0.08 -2.11 1,231,653 45,743 3.68 1.12
6+
2016-06-23 3.78 3.84 3.74 3.80 -0.02 -0.52 1,068,720 40,446 2.62 0.98
7+
2016-06-22 3.75 3.82 3.73 3.82 0.02 0.53 805,845 30,490 2.37 0.74
8+
2016-06-21 3.93 3.93 3.76 3.80 -0.10 -2.56 1,555,987 59,686 4.36 1.42
9+
2016-06-20 4.05 4.05 3.86 3.90 -0.13 -3.23 1,127,471 44,050 4.71 1.03
10+
2016-06-17 3.94 4.06 3.92 4.03 0.08 2.03 692,888 27,781 3.54 0.72
11+
2016-06-16 3.91 3.96 3.87 3.95 0.04 1.02 483,863 19,004 2.30 0.51
12+
2016-06-15 3.84 3.93 3.82 3.91 0.05 1.30 408,695 15,941 2.85 0.43
13+
2016-06-14 3.85 3.88 3.84 3.86 0.00 0.00 290,349 11,203 1.04 0.30
14+
2016-06-13 3.98 4.02 3.84 3.86 -0.16 -3.98 515,692 20,273 4.48 0.54
15+
2016-06-08 4.09 4.09 3.98 4.02 -0.09 -2.19 648,312 26,086 2.68 0.68
16+
2016-06-07 4.07 4.15 4.04 4.11 0.05 1.23 714,603 29,297 2.71 0.75
17+
2016-06-06 4.07 4.10 4.03 4.06 0.00 0.00 336,603 13,668 1.72 0.35
18+
2016-06-03 4.06 4.07 4.04 4.06 -0.01 -0.25 419,868 17,014 0.74 0.44
19+
2016-06-02 4.06 4.11 4.04 4.07 -0.01 -0.25 497,364 20,239 1.72 0.52
20+
2016-06-01 4.08 4.14 4.06 4.08 -0.03 -0.73 746,432 30,543 1.95 0.78
21+
2016-05-31 3.97 4.13 3.97 4.11 0.12 3.01 1,036,674 41,974 4.01 1.08
22+
2016-05-30 3.98 4.01 3.92 3.99 0.02 0.50 401,456 15,906 2.27 0.42
23+
2016-05-27 3.95 4.05 3.92 3.97 -0.01 -0.25 386,450 15,283 3.27 0.40
24+
2016-05-26 3.87 4.00 3.81 3.98 0.11 2.84 595,067 23,245 4.91 0.62
25+
2016-05-25 3.91 3.92 3.86 3.87 -0.01 -0.26 170,958 6,659 1.55 0.18
26+
2016-05-24 3.92 3.92 3.86 3.88 -0.05 -1.27 202,305 7,857 1.53 0.21
27+
2016-05-23 3.93 3.97 3.91 3.93 0.01 0.26 270,932 10,651 1.53 0.28
28+
2016-05-20 3.86 3.94 3.83 3.92 0.00 0.00 321,728 12,525 2.81 0.34
29+
2016-05-19 3.86 4.00 3.85 3.92 0.06 1.55 414,433 16,245 3.89 0.43
30+
2016-05-18 3.93 3.95 3.83 3.86 -0.11 -2.77 359,409 13,913 3.02 0.38
31+
2016-05-17 3.95 4.04 3.94 3.97 0.02 0.51 430,389 17,187 2.53 0.45
32+
2016-05-16 3.89 3.95 3.85 3.95 0.06 1.54 283,243 11,080 2.57 0.30
33+
2016-05-13 3.91 3.95 3.87 3.89 -0.05 -1.27 290,699 11,358 2.03 0.30
34+
2016-05-12 3.88 3.96 3.83 3.94 0.02 0.51 439,001 17,054 3.32 0.46
35+
2016-05-11 3.98 3.98 3.89 3.92 -0.03 -0.76 289,021 11,387 2.28 0.30
36+
2016-05-10 3.90 3.99 3.90 3.95 0.05 1.28 422,443 16,621 2.31 0.44
37+
2016-05-09 4.07 4.07 3.87 3.90 -0.18 -4.41 689,400 27,204 4.90 0.72
38+
2016-05-06 4.26 4.29 4.04 4.08 -0.19 -4.45 655,181 27,400 5.85 0.68
39+
2016-05-05 4.26 4.28 4.23 4.27 0.00 0.00 305,033 12,992 1.17 0.32
40+
2016-05-04 4.28 4.30 4.24 4.27 -0.03 -0.70 428,521 18,292 1.40 0.45
41+
2016-05-03 4.23 4.30 4.19 4.30 0.09 2.14 513,738 21,922 2.61 0.54
42+
2016-04-29 4.22 4.27 4.18 4.21 -0.01 -0.24 374,543 15,832 2.13 0.39
43+
2016-04-28 4.36 4.38 4.16 4.22 -0.12 -2.77 753,752 32,065 5.07 0.79
44+
2016-04-27 4.43 4.44 4.34 4.34 -0.11 -2.47 509,351 22,309 2.25 0.53
45+
2016-04-26 4.41 4.47 4.32 4.45 0.05 1.14 685,285 30,003 3.41 0.72
46+
2016-04-25 4.46 4.46 4.33 4.40 -0.06 -1.35 588,039 25,848 2.91 0.61
47+
2016-04-22 4.56 4.58 4.38 4.46 -0.18 -3.88 1,165,099 52,105 4.31 1.22
48+
2016-04-21 4.56 4.76 4.55 4.64 0.08 1.75 1,941,521 90,364 4.61 2.03
49+
2016-04-20 4.65 4.66 4.37 4.56 -0.05 -1.08 1,132,007 51,340 6.29 1.18
50+
2016-04-19 4.60 4.66 4.54 4.61 0.01 0.22 728,478 33,531 2.61 0.76
51+
2016-04-18 4.55 4.66 4.49 4.60 0.03 0.66 748,206 34,069 3.72 0.78
52+
2016-04-15 4.64 4.64 4.52 4.57 -0.06 -1.30 678,897 31,034 2.59 0.71
53+
2016-04-14 4.66 4.68 4.58 4.63 -0.02 -0.43 825,791 38,208 2.15 0.86
54+
2016-04-13 4.62 4.89 4.61 4.65 0.11 2.42 2,314,881 109,558 6.17 2.42
55+
2016-04-12 4.59 4.60 4.51 4.54 -0.05 -1.09 502,252 22,811 1.96 0.52
56+
2016-04-11 4.56 4.64 4.54 4.59 0.07 1.55 1,376,315 63,359 2.21 1.44
57+
2016-04-08 4.53 4.58 4.45 4.52 0.00 0.00 620,698 27,964 2.88 0.65
58+
2016-04-07 4.62 4.70 4.52 4.52 -0.12 -2.59 662,939 30,397 3.88 0.69
59+
2016-04-06 4.58 4.78 4.56 4.64 0.04 0.87 901,321 41,743 4.78 0.94
60+
2016-04-05 4.49 4.64 4.46 4.60 0.11 2.45 1,392,204 63,528 4.01 1.45
61+
2016-04-01 4.38 4.51 4.37 4.49 0.10 2.28 797,275 35,439 3.19 0.83

‎by-bs4/shares/601857中国石油/Y2016S2.txt‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
2016-06-30 7.22 7.24 7.21 7.23 0.01 0.14 165,720 11,977 0.42 0.01
2+
2016-06-29 7.19 7.23 7.18 7.22 0.03 0.42 172,092 12,396 0.70 0.01
3+
2016-06-28 7.13 7.19 7.12 7.19 0.05 0.70 174,337 12,487 0.98 0.01
4+
2016-06-27 7.08 7.16 7.08 7.14 0.02 0.28 154,911 11,045 1.12 0.01
5+
2016-06-24 7.19 7.21 7.08 7.12 -0.06 -0.84 291,304 20,767 1.81 0.02
6+
2016-06-23 7.22 7.23 7.16 7.18 -0.04 -0.55 117,768 8,473 0.97 0.01
7+
2016-06-22 7.19 7.22 7.18 7.22 0.03 0.42 101,384 7,303 0.56 0.01
8+
2016-06-21 7.21 7.25 7.18 7.19 -0.02 -0.28 130,183 9,393 0.97 0.01
9+
2016-06-20 7.20 7.22 7.17 7.21 0.01 0.14 100,365 7,222 0.69 0.01
10+
2016-06-17 7.17 7.25 7.16 7.20 0.03 0.42 186,709 13,448 1.26 0.01
11+
2016-06-16 7.18 7.21 7.15 7.17 -0.05 -0.69 187,591 13,462 0.83 0.01
12+
2016-06-15 7.16 7.23 7.13 7.22 0.03 0.42 154,929 11,146 1.39 0.01
13+
2016-06-14 7.19 7.21 7.17 7.19 0.00 0.00 125,601 9,023 0.56 0.01
14+
2016-06-13 7.26 7.39 7.18 7.19 -0.11 -1.51 242,593 17,605 2.88 0.02
15+
2016-06-08 7.32 7.32 7.27 7.30 -0.01 -0.14 130,984 9,563 0.68 0.01
16+
2016-06-07 7.35 7.35 7.31 7.33 -0.01 -0.14 79,628 5,837 0.55 0.00
17+
2016-06-06 7.35 7.36 7.32 7.34 -0.02 -0.27 105,351 7,726 0.54 0.01
18+
2016-06-03 7.34 7.38 7.30 7.36 0.03 0.41 246,468 18,097 1.09 0.02
19+
2016-06-02 7.33 7.34 7.30 7.33 0.01 0.14 116,684 8,542 0.55 0.01
20+
2016-06-01 7.36 7.37 7.32 7.32 -0.05 -0.68 225,882 16,582 0.68 0.01
21+
2016-05-31 7.22 7.38 7.21 7.37 0.14 1.94 395,318 28,893 2.35 0.02
22+
2016-05-30 7.22 7.24 7.17 7.23 0.03 0.42 133,729 9,650 0.97 0.01
23+
2016-05-27 7.21 7.22 7.17 7.20 -0.02 -0.28 90,752 6,531 0.69 0.01
24+
2016-05-26 7.21 7.25 7.15 7.22 0.03 0.42 143,016 10,305 1.39 0.01
25+
2016-05-25 7.22 7.24 7.19 7.19 -0.03 -0.42 111,170 8,018 0.69 0.01
26+
2016-05-24 7.19 7.22 7.15 7.22 0.02 0.28 149,170 10,711 0.97 0.01
27+
2016-05-23 7.21 7.23 7.19 7.20 -0.01 -0.14 105,591 7,613 0.55 0.01
28+
2016-05-20 7.17 7.21 7.15 7.21 0.03 0.42 87,156 6,262 0.84 0.01
29+
2016-05-19 7.18 7.23 7.15 7.18 -0.02 -0.28 169,519 12,204 1.11 0.01
30+
2016-05-18 7.21 7.22 7.11 7.20 -0.04 -0.55 288,083 20,628 1.52 0.02
31+
2016-05-17 7.21 7.25 7.20 7.24 0.04 0.56 178,868 12,926 0.69 0.01
32+
2016-05-16 7.17 7.20 7.12 7.20 0.01 0.14 132,203 9,475 1.11 0.01
33+
2016-05-13 7.20 7.25 7.16 7.19 -0.01 -0.14 210,406 15,135 1.25 0.01
34+
2016-05-12 7.21 7.21 7.14 7.20 -0.02 -0.28 172,218 12,349 0.97 0.01
35+
2016-05-11 7.21 7.25 7.18 7.22 0.03 0.42 148,519 10,720 0.97 0.01
36+
2016-05-10 7.18 7.22 7.16 7.19 -0.01 -0.14 137,132 9,860 0.83 0.01
37+
2016-05-09 7.32 7.32 7.18 7.20 -0.14 -1.91 347,541 25,119 1.91 0.02
38+
2016-05-06 7.46 7.46 7.34 7.34 -0.12 -1.61 327,262 24,187 1.61 0.02
39+
2016-05-05 7.45 7.48 7.42 7.46 0.00 0.00 177,455 13,212 0.80 0.01
40+
2016-05-04 7.48 7.51 7.43 7.46 -0.05 -0.67 213,059 15,899 1.07 0.01
41+
2016-05-03 7.40 7.51 7.39 7.51 0.09 1.21 292,456 21,831 1.62 0.02
42+
2016-04-29 7.45 7.49 7.38 7.42 -0.11 -1.46 374,143 27,800 1.46 0.02
43+
2016-04-28 7.54 7.58 7.47 7.53 0.00 0.00 222,569 16,736 1.46 0.01
44+
2016-04-27 7.56 7.59 7.52 7.53 -0.01 -0.13 208,058 15,723 0.93 0.01
45+
2016-04-26 7.58 7.58 7.49 7.54 -0.02 -0.26 180,850 13,631 1.19 0.01
46+
2016-04-25 7.55 7.58 7.47 7.56 -0.02 -0.26 204,615 15,397 1.45 0.01
47+
2016-04-22 7.53 7.63 7.50 7.58 0.01 0.13 251,020 18,971 1.72 0.02
48+
2016-04-21 7.52 7.64 7.52 7.57 -0.02 -0.26 361,312 27,424 1.58 0.02
49+
2016-04-20 7.65 7.69 7.43 7.59 -0.06 -0.78 635,918 47,904 3.40 0.04
50+
2016-04-19 7.63 7.66 7.56 7.65 0.06 0.79 275,055 20,920 1.32 0.02
51+
2016-04-18 7.68 7.68 7.56 7.59 -0.16 -2.06 343,820 26,132 1.55 0.02
52+
2016-04-15 7.80 7.80 7.72 7.75 -0.02 -0.26 243,138 18,841 1.03 0.02
53+
2016-04-14 7.84 7.85 7.74 7.77 -0.02 -0.26 402,957 31,366 1.41 0.02
54+
2016-04-13 7.70 7.84 7.69 7.79 0.14 1.83 858,613 66,793 1.96 0.05
55+
2016-04-12 7.65 7.68 7.62 7.65 0.01 0.13 360,600 27,586 0.79 0.02
56+
2016-04-11 7.56 7.66 7.55 7.64 0.12 1.60 514,811 39,261 1.46 0.03
57+
2016-04-08 7.58 7.59 7.49 7.52 -0.09 -1.18 385,975 29,092 1.31 0.02
58+
2016-04-07 7.72 7.73 7.58 7.61 -0.09 -1.17 484,217 37,036 1.95 0.03
59+
2016-04-06 7.64 7.83 7.63 7.70 0.04 0.52 839,982 64,903 2.61 0.05
60+
2016-04-05 7.61 7.68 7.56 7.66 0.00 0.00 493,144 37,627 1.57 0.03
61+
2016-04-01 7.58 7.68 7.51 7.66 0.05 0.66 574,414 43,637 2.23 0.04

‎by-bs4/shares/Y2016S3.txt‎

Lines changed: 0 additions & 64 deletions
This file was deleted.

‎by-bs4/shares/Y2016S4.txt‎

Lines changed: 0 additions & 23 deletions
This file was deleted.

‎by-bs4/shares/zhongshiyou.py‎

Lines changed: 28 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,40 @@
11
import requests
22
from bs4 import BeautifulSoup
3-
4-
url = 'http://quotes.money.163.com/trade/lsjysj_601857.html?year=2016&season=3'
3+
import os
54

65
headers = {
76
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.1708.400 QQBrowser/9.5.9635.400'
87
}
98

10-
data = requests.get(url , headers=headers)
11-
soup = BeautifulSoup(data.text, 'lxml')
12-
# print soup
13-
date = soup.select('div.inner_box > table > tr > td')
9+
url = 'http://quotes.money.163.com/trade/lsjysj_601857.html?year=2016&season=4'
10+
11+
def sharesCrawl(shareCode,year,month):
12+
13+
url = 'http://quotes.money.163.com/trade/lsjysj_'+shareCode+'.html?year='+year+'&season='+month
14+
15+
data = requests.get(url, headers=headers)
16+
soup = BeautifulSoup(data.text, 'lxml')
17+
# print soup
18+
title = soup.select('h1.name > a')[0].get_text()
19+
20+
date = soup.select('div.inner_box > table > tr > td')
21+
22+
if os.path.exists('./'+shareCode+title) == False:
23+
#create the share folder
24+
os.mkdir('./'+shareCode+title)
25+
26+
f = open('./'+shareCode+title+'/Y'+year+'S'+month+'.txt','wb')
27+
for index,value in enumerate(date):
28+
if index % 11 == 10:
29+
f.write(value.get_text()+'\n')
30+
else:
31+
f.write(value.get_text() +'\t')
32+
f.close()
33+
34+
sharesCrawl('601600','2016','2')
1435

15-
f = open('./Y2016S3.txt','wb')
16-
for index,value in enumerate(date):
17-
print value.get_text(),index
18-
if index % 11 == 10 :
19-
f.write(value.get_text()+'\n')
20-
else:
21-
f.write(value.get_text() +'\t')
22-
f.close()
2336

2437
"""
2538
body > div.area > div.inner_box > table > tbody > tr:nth-child(1) > td:nth-child(1)
39+
body > div.area > div.header > div.stock_info > table > tbody > tr > td.col_1 > h1 > a
2640
"""

‎by-scarpy/wikiSpider/wikiSpider/pipelines.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,5 @@
88

99
class WikispiderPipeline(object):
1010
def process_item(self, item, spider):
11+
print item['title']
1112
return item

‎by-scarpy/wikiSpider/wikiSpider/settings.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,9 @@
6464

6565
# Configure item pipelines
6666
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
67-
#ITEM_PIPELINES = {
68-
# 'wikiSpider.pipelines.SomePipeline': 300,
69-
#}
67+
ITEM_PIPELINES = {
68+
'wikiSpider.pipelines.WikispiderPipeline': 300,
69+
}
7070

7171
# Enable and configure the AutoThrottle extension (disabled by default)
7272
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html

‎by-scarpy/wikiSpider/wikiSpider/settings.pyc‎

81 Bytes
Binary file not shown.

‎by-scarpy/wikiSpider/wikiSpider/spiders/csdn.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# -*- coding: utf-8 -*-
2+
import scrapy
3+
4+
5+
class CsdnSpider(scrapy.Spider):
6+
name = "csdn"
7+
allowed_domains = ["csdn.net"]
8+
start_urls = (
9+
'http://blog.csdn.net/',
10+
)
11+
12+
def parse(self, response):
13+
pass

0 commit comments

Comments
 (0)