Migrate to Stretch

MaxSem · MaxSem · commit 9c050dda2111 · 2019-03-27T17:20:47.000-07:00
* Create a requirements file * Use virtualenv * Switch to pymysql for DB access. Bug: https://phabricator.wikimedia.org/T218747
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,5 @@
 .config.py
-
 .DS_STORE
-
 *.pyc
+venv
+config.py
diff --git a/README.md b/README.md
@@ -1,9 +1,9 @@
 # Database Reports
-Generates statistical reports which are used by community members to improve Wikipedia. 
+Generates statistical reports which are used by community members to improve Wikipedia.
 
-This project allows the [Community Tech bot](https://en.wikipedia.org/wiki/User:Community_Tech_bot) to make periodic updates to these reports on different language Wikipedias. As of now the project support report generation for English ([see here](https://en.wikipedia.org/wiki/Wikipedia:Database_reports)), Vietnamese, Korean and French Wikipedia. 
+This project allows the [Community Tech bot](https://en.wikipedia.org/wiki/User:Community_Tech_bot) to make periodic updates to these reports on different language Wikipedias. As of now the project support report generation for English ([see here](https://en.wikipedia.org/wiki/Wikipedia:Database_reports)), Vietnamese, Korean and French Wikipedia.
 
-## Specific statistics that the reports support: 
+## Specific statistics that the reports support:
 * Unused templates
 * Forgotten articles
 * Most used templates
@@ -13,21 +13,30 @@ This project allows the [Community Tech bot](https://en.wikipedia.org/wiki/User:
 * Unused file redirects
 * Forgotten articles
 * Page with most revisions
-* Page count by namespace 
-* Most edited articles last month 
+* Page count by namespace
+* Most edited articles last month
 * PRODed articles with deletion logs
 * Editors eligible for autopatrol privileges
 * Active editors with the longest-established accounts
 
-## Generating a report 
+## Installation
+Virtualenv is recommended:
+
+    virtualenv venv
+    source venv/bin/activate
+    pip install -r requirements.txt
+
+After installation, either activate virtualenv like above or use `venv/bin/python` to run scripts.
+
+## Generating a report
 * Log into the Toolforge bastion using your Wikimedia developer account ```ssh username@login.tools.wmflabs.org```
 * Become your tool account ```become database-reports```
-* Run ```python main.py test articles_by_size```. It takes two arguments; in this example test refers to `test.wikipedia.org` and `articles_by_size` is the type of statistics you're requesting. This command outputs the name of the page on which the report got dumped  
+* Run ```python main.py test articles_by_size```. It takes two arguments; in this example test refers to `test.wikipedia.org` and `articles_by_size` is the type of statistics you're requesting. This command outputs the name of the page on which the report got dumped
 * To alter the default settings for periodic updates, make changes to the crontab file ```crontab -e```
 
 ## Adding support for a report
-* To add support for a specific statistics that you would like to see in a report, declare a function in `main.py` and define it in `reports.py` 
-* To provide support for translations in a specific language, include the dictionary in `i18n/i18n.py` 	
+* To add support for a specific statistics that you would like to see in a report, declare a function in `main.py` and define it in `reports.py`
+* To provide support for translations in a specific language, include the dictionary in `i18n/i18n.py`
 
 ## Contributing
 Bug reports, fixes, and new features are welcomed. If you'd like to contribute code please:
diff --git a/main.py b/main.py
@@ -1,5 +1,5 @@
 import mwclient
-import MySQLdb
+import pymysql
 from displayTable import *
 from config import *
 from reports import *
@@ -18,7 +18,7 @@ def main ( args ):
 
 class Run:
 	def __init__( self, wiki ):
-		self.db = MySQLdb.connect( host = wiki + 'wiki.labsdb', user = credentials['user'], passwd = credentials['pass'], db = wiki + 'wiki_p' )
+		self.db = pymysql.connect( host = wiki + 'wiki.labsdb', user = credentials['user'], passwd = credentials['pass'], db = wiki + 'wiki_p' )
 		self.site = mwclient.Site( wiki + '.wikipedia.org' )
 		self.site.login( cttbot['user'], cttbot['pass'] )
 		self.rep = Reports( self.site, self.db, wiki )
diff --git a/reports.py b/reports.py
@@ -1,5 +1,5 @@
 import mwclient
-import MySQLdb
+import pymysql
 from i18n import i18n
 import datetime
 from displayTable import *
@@ -13,6 +13,8 @@ def __init__( self, site, db, wiki ):
 		self.db = db
 		self.site = site
 		self.wiki = wiki
+		# 30s is not enough
+		self.site.requests['timeout'] = 120
 
 	# Oldest edited articles
 	# Run time on enwiki 5 hours 23 minutes as of 8 Sept 2015
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,2 @@
+mwclient == 0.9.3
+PyMySQL == 0.9.3
diff --git a/test.py b/test.py
@@ -1,5 +1,5 @@
 import mwclient
-import MySQLdb
+import pymysql
 import datetime
 from .config import *
 from reports import *
@@ -8,7 +8,7 @@
 
 def forgotten_articles():
 
-	db = MySQLdb.connect( host = 'enwiki.labsdb', user = credentials['user'], passwd = credentials['pass'], db = 'enwiki_p' )
+	db = pymysql.connect( host = 'enwiki.labsdb', user = credentials['user'], passwd = credentials['pass'], db = 'enwiki_p' )
 	site = mwclient.Site( 'en.wikipedia.org' )
 	site.login( cttbot['user'], cttbot['pass'] )
 	print 'initiated'

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+mwclient == 0.9.3`
	`2`	`+PyMySQL == 0.9.3`