Java-Jersey Python-Flask
Max Lai
2018/06/01
PyCon Taiwan 2018
About Me
• Taichung.py
• Agile Taichung
• Agile Tour Taichung 2016~2018
• (Jarvish Inc.)
PyCon Taiwan 2018
Agenda
•
• RESTful Framework
• Flask vs Django REST framework
• SQLAlchemy MySQL DB
• API Documentation by Swagger
• RESTful API
• Tavern RESTful API
• (Canary Release) API A/B test
• Q&A
PyCon Taiwan 2018
• http://bit.ly/pycontw2018-flaskSlides
• https://github.com/cclai999/pycontw2018-demoCode
• http://bit.ly/2kAIgMoDemo
PyCon Taiwan 2018
• Jarvish AI
• back end lead Azure DevOps
• back end Java-Jersey, Scala-Akka
Project
PyCon Taiwan 2018
) (
PyCon Taiwan 2018
X
Before:
PyCon Taiwan 2018
Nginx
Java/Jersey
Scala/Akka
Redis
MySQL
RESTful API
API Set-A
API Set-B
Ongoing:
PyCon Taiwan 2018
Nginx
Scala/Akka
Redis
MySQL
RESTful API
API Set-A
API Set-B
Python/Flask
API Set-C
Python/Flask
After: ( )
PyCon Taiwan 2018
Nginx
Redis
MySQL
Python/Flask
Python/Flask
Docker
RESTful API
Python/Flask
RESTful Framework
Flask vs. Django
• Flask is light, with extensions
• Django is large, batteries included
• Both are good
• Which one do I choose?
• depend on the context
PyCon Taiwan 2018
Hello World in Flask
PyCon Taiwan 2018
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello():
return 'Hello World!'
$ FLASK_APP=hello.py flask run
#start flask server
#web server in a file
Hello World in Django
PyCon Taiwan 2018
$ django-admin startproject mytsite
$ cd mysite
$ python manage.py startapp hello
# edit mysite/settings.py
INSTALLED_APPS = [
...
'hello',
]
# creating project hello
Hello World in Django
PyCon Taiwan 2018
# edit hello/views.py
from django.http import HttpResponse
def index(request):
return HttpResponse("Hello World!")
Hello World in Django
PyCon Taiwan 2018
$ python manage.py runserver
# edit mysite/urls.py
from django.conf.urls import include, url
from hello import views
urlpatterns = [
url(r'^$', views.hello),
]
#start web server
RESTful API
PyCon Taiwan 2018
HTTP Method Action Examples
GET
Obtain information about a
resource
http://[hostname]/todo/api/v1.0/tasks
(Retrieve list of tasks)
GET
Obtain information about a
resource
http://[hostname]/todo/api/v1.0/tasks/1
(Retrieve task #1)
POST Create a new resource
http://[hostname]/todo/api/v1.0/tasks
(Create a new task, from data provided with the request)
PUT Update a resource
http://[hostname]/todo/api/v1.0/tasks/1
(update task #1, from data provided with the request)
DELETE Delete a resource
http://[hostname]/todo/api/v1.0/tasks/1
(delete task #1)
API in Flask
PyCon Taiwan 2018
from flask import jsonify
@app.route('/todo/api/v1.0/tasks/<int:task_id>', methods = ['GET'])
def get_task(task_id):
task = filter(lambda t: t['id'] == task_id, tasks)
if len(task) == 0:
abort(404)
return jsonify( { 'task': make_public_task(task[0]) } )
http://localhost:5000/todo/api/v1.0/tasks/1
Serialization: Flask-Marshmallow with SQLAlchemy, MongoEngine, etc
API in Django
• Django REST Framework is great!
• Browsable API: documents API with a human-friendly HTML output
• Auth Support: rich support for various authentication protocols along
with permissions and throttling policies
• Serializers: converting querysets/instances to native Python datatypes
that can be easily rendered into JSON and XML
• Throttling: generally used for rate limiting API requests from a single
user
• It is subject to same restrictions as Django
PyCon Taiwan 2018
RESTful API
API biz logic
Template, Admin
PyCon Taiwan 2018
Flask MySQL DB
PyCon Taiwan 2018
Flask MySQL Schema
PyCon Taiwan 2018
mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb"
engine = create_engine(mysql_db_url, echo=False,
pool_recycle=3600)
Base = declarative_base()
Base.metadata.reflect(engine)
db_session = scoped_session(sessionmaker(bind=engine))
class Todo_Tasks(Base):
__table__ = Base.metadata.tables['todo_tasks']
Flask-SQLAlchemy
(OperationalError)
(2006, 'MySQL server has gone away')
PyCon Taiwan 2018
MySQL features an automatic connection close behavior,
for connections that have been idle for a fixed period of time,
defaulting to eight hours.
source: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=mysql
1
• pool recycle . ex. 3600 sec.
PyCon Taiwan 2018
mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb"
engine = create_engine(mysql_db_url, pool_recycle=3600)
source: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=mysql
‘MySQL server has gone away’
2
connection test, “SELECT 1”
PyCon Taiwan 2018
source: http://docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects
@event.listens_for(engine, "engine_connect")
def ping_connection(connection, branch):
if branch:
return
save_should_close_with_result = connection.should_close_with_result
connection.should_close_with_result = False
try:
connection.scalar(select([1]))
except exc.DBAPIError as err:
if err.connection_invalidated:
connection.scalar(select([1]))
else:
raise
finally:
connection.should_close_with_result = save_should_close_with_result
2(cont.)
• SQLAlchemy v1.2
PyCon Taiwan 2018
source: http://docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects
mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb"
engine = create_engine(mysql_db_url, pool_recycle=3600, pool_pre_ping=True)
API
documentation
by Swagger
• API
•
PyCon Taiwan 2018
API documentation by Flasgger(Swagger)
• Swagger API HTML Javascript
• OpenAPI Specification ( Swagger Specification)
RESTful API
• JSON or YAML API spec Flasgger
SwaggerUI
• Marshmallow APISpec Flasgger specs base
template
• Demo http://flasgger.pythonanywhere.com/
PyCon Taiwan 2018
• Martin Fowler
• infrastructure level
PyCon Taiwan 2018
• ( )
•
•
• API
•
•
PyCon Taiwan 2018
Nginx
Java/API Set-ARedis
MySQL
API
Flask/API Set-A
API
Why Tavern
• Why not Postman, Insomnia?
• focus on automated testing of APIs
• test syntax is more intuitive (YAML format)
• easy to write for most people
• Why not pyresttest?
• it is no longer actively developed.
PyCon Taiwan 2018
API returns the double of a number
PyCon Taiwan 2018
@app.route("/double", methods=["POST"])
def double_number():
r = request.get_json()
try:
number = r["number"]
except (KeyError, TypeError):
return jsonify({"error": "no number passed"}), 400
try:
double = int(number)*2
except ValueError:
return jsonify({"error": "a number was not passed"}), 400
return jsonify({"double": double}), 200
Tavern RESTful API
PyCon Taiwan 2018
test_name: Make sure server doubles number properly
stages:
- name: Make sure number is returned correctly
request:
url: http://localhost:5000/double
json:
number: 5
method: POST
headers:
content-type: application/json
response:
status_code: 200
body:
double: 10
Tavern RESTful API
PyCon Taiwan 2018
(Canary Release)1
• Canary
•
• 17
•
PyCon Taiwan 2018
(Canary Release)2
PyCon Taiwan 2018
source: http://bit.ly/2seHrwg
(Canary Release)3
PyCon Taiwan 2018
source: http://bit.ly/2seHrwg
(Canary Release)4
PyCon Taiwan 2018
source: http://bit.ly/2seHrwg
Nginx Upstream
PyCon Taiwan 2018
Nginx
Java/API Set-ARedis
MySQL Flask/API Set-A
80%
20%
Nginx
PyCon Taiwan 2018
http {
upstream demoweb {
server web1:8000; # demo web server1
server web2:8000 weight=4; # demo web server2
}
location ~ ^/demo/api/v1.0/servername/(.*)$ {
proxy_pass http://demoweb/demo/api/v1.0/servername/$1;
# Redefine the header fields that NGINX sends to the upstream
server
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
……
}
}
Demo
PyCon Taiwan 2018
Takeaways
PyCon Taiwan 2018
Flask
Taichung.py Agile.Taichung
PyCon Taiwan 2018

Java-Jersey 到 Python-Flask 服務不中斷重構之旅

  • 1.
  • 2.
    About Me • Taichung.py •Agile Taichung • Agile Tour Taichung 2016~2018 • (Jarvish Inc.) PyCon Taiwan 2018
  • 3.
    Agenda • • RESTful Framework •Flask vs Django REST framework • SQLAlchemy MySQL DB • API Documentation by Swagger • RESTful API • Tavern RESTful API • (Canary Release) API A/B test • Q&A PyCon Taiwan 2018
  • 4.
  • 5.
    • Jarvish AI •back end lead Azure DevOps • back end Java-Jersey, Scala-Akka Project PyCon Taiwan 2018
  • 6.
  • 7.
  • 8.
    Ongoing: PyCon Taiwan 2018 Nginx Scala/Akka Redis MySQL RESTfulAPI API Set-A API Set-B Python/Flask API Set-C Python/Flask
  • 9.
    After: ( ) PyConTaiwan 2018 Nginx Redis MySQL Python/Flask Python/Flask Docker RESTful API Python/Flask
  • 10.
    RESTful Framework Flask vs.Django • Flask is light, with extensions • Django is large, batteries included • Both are good • Which one do I choose? • depend on the context PyCon Taiwan 2018
  • 11.
    Hello World inFlask PyCon Taiwan 2018 from flask import Flask app = Flask(__name__) @app.route('/') def hello(): return 'Hello World!' $ FLASK_APP=hello.py flask run #start flask server #web server in a file
  • 12.
    Hello World inDjango PyCon Taiwan 2018 $ django-admin startproject mytsite $ cd mysite $ python manage.py startapp hello # edit mysite/settings.py INSTALLED_APPS = [ ... 'hello', ] # creating project hello
  • 13.
    Hello World inDjango PyCon Taiwan 2018 # edit hello/views.py from django.http import HttpResponse def index(request): return HttpResponse("Hello World!")
  • 14.
    Hello World inDjango PyCon Taiwan 2018 $ python manage.py runserver # edit mysite/urls.py from django.conf.urls import include, url from hello import views urlpatterns = [ url(r'^$', views.hello), ] #start web server
  • 15.
    RESTful API PyCon Taiwan2018 HTTP Method Action Examples GET Obtain information about a resource http://[hostname]/todo/api/v1.0/tasks (Retrieve list of tasks) GET Obtain information about a resource http://[hostname]/todo/api/v1.0/tasks/1 (Retrieve task #1) POST Create a new resource http://[hostname]/todo/api/v1.0/tasks (Create a new task, from data provided with the request) PUT Update a resource http://[hostname]/todo/api/v1.0/tasks/1 (update task #1, from data provided with the request) DELETE Delete a resource http://[hostname]/todo/api/v1.0/tasks/1 (delete task #1)
  • 16.
    API in Flask PyConTaiwan 2018 from flask import jsonify @app.route('/todo/api/v1.0/tasks/<int:task_id>', methods = ['GET']) def get_task(task_id): task = filter(lambda t: t['id'] == task_id, tasks) if len(task) == 0: abort(404) return jsonify( { 'task': make_public_task(task[0]) } ) http://localhost:5000/todo/api/v1.0/tasks/1 Serialization: Flask-Marshmallow with SQLAlchemy, MongoEngine, etc
  • 17.
    API in Django •Django REST Framework is great! • Browsable API: documents API with a human-friendly HTML output • Auth Support: rich support for various authentication protocols along with permissions and throttling policies • Serializers: converting querysets/instances to native Python datatypes that can be easily rendered into JSON and XML • Throttling: generally used for rate limiting API requests from a single user • It is subject to same restrictions as Django PyCon Taiwan 2018
  • 18.
    RESTful API API bizlogic Template, Admin PyCon Taiwan 2018
  • 19.
  • 20.
    Flask MySQL Schema PyConTaiwan 2018 mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb" engine = create_engine(mysql_db_url, echo=False, pool_recycle=3600) Base = declarative_base() Base.metadata.reflect(engine) db_session = scoped_session(sessionmaker(bind=engine)) class Todo_Tasks(Base): __table__ = Base.metadata.tables['todo_tasks']
  • 21.
    Flask-SQLAlchemy (OperationalError) (2006, 'MySQL serverhas gone away') PyCon Taiwan 2018 MySQL features an automatic connection close behavior, for connections that have been idle for a fixed period of time, defaulting to eight hours. source: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=mysql
  • 22.
    1 • pool recycle. ex. 3600 sec. PyCon Taiwan 2018 mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb" engine = create_engine(mysql_db_url, pool_recycle=3600) source: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=mysql ‘MySQL server has gone away’
  • 23.
    2 connection test, “SELECT1” PyCon Taiwan 2018 source: http://docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects @event.listens_for(engine, "engine_connect") def ping_connection(connection, branch): if branch: return save_should_close_with_result = connection.should_close_with_result connection.should_close_with_result = False try: connection.scalar(select([1])) except exc.DBAPIError as err: if err.connection_invalidated: connection.scalar(select([1])) else: raise finally: connection.should_close_with_result = save_should_close_with_result
  • 24.
    2(cont.) • SQLAlchemy v1.2 PyConTaiwan 2018 source: http://docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects mysql_db_url="mysql://maxlai:pycontw2018@127.0.0.1/demodb" engine = create_engine(mysql_db_url, pool_recycle=3600, pool_pre_ping=True)
  • 25.
  • 26.
    API documentation byFlasgger(Swagger) • Swagger API HTML Javascript • OpenAPI Specification ( Swagger Specification) RESTful API • JSON or YAML API spec Flasgger SwaggerUI • Marshmallow APISpec Flasgger specs base template • Demo http://flasgger.pythonanywhere.com/ PyCon Taiwan 2018
  • 27.
    • Martin Fowler •infrastructure level PyCon Taiwan 2018
  • 28.
    • ( ) • • •API • • PyCon Taiwan 2018 Nginx Java/API Set-ARedis MySQL API Flask/API Set-A API
  • 29.
    Why Tavern • Whynot Postman, Insomnia? • focus on automated testing of APIs • test syntax is more intuitive (YAML format) • easy to write for most people • Why not pyresttest? • it is no longer actively developed. PyCon Taiwan 2018
  • 30.
    API returns thedouble of a number PyCon Taiwan 2018 @app.route("/double", methods=["POST"]) def double_number(): r = request.get_json() try: number = r["number"] except (KeyError, TypeError): return jsonify({"error": "no number passed"}), 400 try: double = int(number)*2 except ValueError: return jsonify({"error": "a number was not passed"}), 400 return jsonify({"double": double}), 200
  • 31.
    Tavern RESTful API PyConTaiwan 2018 test_name: Make sure server doubles number properly stages: - name: Make sure number is returned correctly request: url: http://localhost:5000/double json: number: 5 method: POST headers: content-type: application/json response: status_code: 200 body: double: 10
  • 32.
  • 33.
    (Canary Release)1 • Canary • •17 • PyCon Taiwan 2018
  • 34.
    (Canary Release)2 PyCon Taiwan2018 source: http://bit.ly/2seHrwg
  • 35.
    (Canary Release)3 PyCon Taiwan2018 source: http://bit.ly/2seHrwg
  • 36.
    (Canary Release)4 PyCon Taiwan2018 source: http://bit.ly/2seHrwg
  • 37.
    Nginx Upstream PyCon Taiwan2018 Nginx Java/API Set-ARedis MySQL Flask/API Set-A 80% 20%
  • 38.
    Nginx PyCon Taiwan 2018 http{ upstream demoweb { server web1:8000; # demo web server1 server web2:8000 weight=4; # demo web server2 } location ~ ^/demo/api/v1.0/servername/(.*)$ { proxy_pass http://demoweb/demo/api/v1.0/servername/$1; # Redefine the header fields that NGINX sends to the upstream server proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; …… } }
  • 39.
  • 40.
  • 41.