Skip to content

Commit 268a54b

Browse files
paulovnpaulovn
committed
* magic processing split out to magics.py
 * new magic: %load <filename> * in %auth magic - can read parameters from environment variables - suppress printing out password
1 parent 6836b3d commit 268a54b

File tree

6 files changed

+368
-244
lines changed

6 files changed

+368
-244
lines changed

‎CHANGES.txt‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
v. 1.3.0
22
* improved error messages on HTTP errors
33
* strip whitespace in all lines, including comment lines (@gpotdevin)
4-
* added new magics: %method (@alexisdimi), %http_header
5-
* added new option `none` to %format magic
4+
* added new magics: %method (@alexisdimi), %http_header, %load
5+
* added new option `none` to %format magic, for manual control of format request
6+
* can read `%auth` parameters from environment variables
7+
* do not print out password for `%auth` magic
8+
* magic processing split out to magics.py
69

710
v. 1.2.1
811
* bugfix (@blake-regalia, @alexisdimi): in label selection for graph nodes, itervalues() does not exist in python3

‎Makefile‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Build the Python source package & upload to PyPi
2+
3+
# Package version: taken from the __init__.py file
4+
PKGNAME_BASE_UND := $(subst -,_,$(PKGNAME_BASE))
5+
VERSION_FILE := sparqlkernel/constants.py
6+
VERSION := $(shell grep __version__ $(VERSION_FILE) | sed -r "s/__version__ = '(.*)'/\1/")
7+
8+
PKG := dist/sparqlkernel-$(VERSION).tar.gz
9+
10+
# ----------------------------------------------------------------------------
11+
12+
all:
13+
python setup.py sdist
14+
15+
clean:
16+
rm -f $(PKG)
17+
18+
install: all
19+
pip install --upgrade $(PKG)
20+
21+
reinstall: clean install
22+
23+
# ----------------------------------------------------------------------------
24+
25+
upload: all
26+
twine upload $(PKG)
27+
# python setup.py sdist --formats=gztar upload
28+
29+
upload-test: all
30+
twine upload -r pypitest $(PKG)
31+
32+

‎doc/magics.rst‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,18 @@ Set logging level. Available levels are: *critical*, *error*, *warning*,
6363
*info*, *debug*.
6464

6565

66+
``%load``
67+
---------
68+
69+
Open a file containing magic lines, read them and process them. Syntax is just::
70+
71+
%load <filename>
72+
73+
and the file can contain only magic lines (full magics, starting with ``%``),
74+
or empty/comment lines.
75+
76+
77+
6678
2. Request creation
6779
===================
6880

@@ -172,6 +184,20 @@ remove a defined authentication, just use::
172184

173185
%auth none
174186

187+
Either of the three components of the authentication (*method*, *user*, *password*)
188+
can be read from environment variables, by using the ``env:`` prefix. E.g.::
189+
190+
%auth basic env:ENDPOINT_USERNAME env:ENDPOINT_PASSWD
191+
192+
will use basic authentication, reading the username from the environment
193+
variable ``ENDPOINT_USERNAME`` and the password from the environment variable
194+
``ENDPOINT_PASSWD``. This allows keeping credentials out of the notebook
195+
(another way would be to use the ``%load`` magic).
196+
197+
Note that, when printing out magic evaluation in the notebook, the password is
198+
never shown.
199+
200+
175201

176202
3. Query formulation
177203
====================
@@ -180,7 +206,7 @@ remove a defined authentication, just use::
180206
``%prefix``
181207
-----------
182208

183-
Set a URI prefix for all subsequent queries. Its syntax is::
209+
Define a URI prefix available for all subsequent queries. Its syntax is::
184210

185211
%prefix <name> <uri>
186212

@@ -199,14 +225,14 @@ Set the default graph for all queries, as::
199225

200226
%graph <uri>
201227

202-
It is equivalent to using the ``FROM`` SPARQL keyword in a query, but when it
203-
is defined is automatically sent in all queries.
228+
It is equivalent to using the ``FROM`` SPARQL keyword in a query, but when
229+
defined it is automatically sent in all queries.
204230

205231

206232
``%header``
207233
-----------
208234

209-
Prepends a certain textual header line to all sparql queries. This can be used
235+
Prepends a certain textual header line to all SPARQL queries. This can be used
210236
to set some (potentially non SPARQL) command in the query.
211237

212238
For instance Virtuoso endpoints accept the *DEFINE* keyword which can be used
@@ -273,9 +299,13 @@ Default is 20. It is also possible to use::
273299
``%lang``
274300
---------
275301

276-
Selects the language chosen for the RDF labels, in either the *table* or the
302+
Selects the language(s) preferred for the RDF labels, in either the *table* or the
277303
*diagram* formats
278304

305+
Syntax is::
306+
307+
%lang <lang> [...] | default | all
308+
279309

280310
``%outfile``
281311
------------

‎sparqlkernel/connection.py‎

Lines changed: 13 additions & 220 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
"""
22
The class used to manage the connection to SPARQL endpoint: send queries and
3-
format results for notebook display. Also process all the defined magics
3+
format results for notebook display.
44
"""
55

66
from __future__ import print_function
@@ -49,39 +49,6 @@
4949
SPARQLWrapper.XML: set(_SPARQL_XML)
5050
}
5151

52-
# ----------------------------------------------------------------------
53-
54-
# The list of implemented magics with their help, as a pair [param,help-text]
55-
magics = {
56-
'%lsmagics': ['', 'list all magics'],
57-
'%endpoint': ['<url>', 'set SPARQL endpoint. **REQUIRED**'],
58-
'%auth': ['(basic|digest|none) <username> <passwd>', 'send HTTP authentication'],
59-
'%qparam': ['<name> [<value>]', 'add (or delete) a persistent custom parameter to all queries'],
60-
'%http_header': ['<name> [<value>]', 'add (or delete) an arbitrary HTTP header to all queries'],
61-
'%prefix': ['<name> [<uri>]', 'set (or delete) a persistent URI prefix for all queries'],
62-
'%header': ['<string> | OFF', 'add a persistent SPARQL header line before all queries, or delete all defined headers'],
63-
'%graph': ['<uri>', 'set default graph for the queries'],
64-
'%format': ['JSON | N3 | XML | default | any | none', 'set requested result format'],
65-
'%display': ['raw | table [withtypes] | diagram [svg|png] [withliterals]',
66-
'set display format'],
67-
'%lang': ['<lang> [...] | default | all',
68-
'language(s) preferred for labels'],
69-
'%show': ['<n> | all',
70-
'maximum number of shown results'],
71-
'%outfile': ['<filename> | off', 'save raw output to a file (use "%d" in name to add cell number, "off" to cancel saving)'],
72-
'%log': ['critical | error | warning | info | debug',
73-
'set logging level'],
74-
'%method': ['get | post', 'set HTTP method'],
75-
}
76-
77-
# The full list of all magics
78-
magic_help = ('Available magics:\n' +
79-
' '.join(sorted(magics.keys())) +
80-
'\n\n' +
81-
'\n'.join(('{0} {1} : {2}'.format(k, *v)
82-
for k, v in sorted(magics.items(), key=itemgetter(0)))))
83-
84-
8552
# ----------------------------------------------------------------------
8653

8754
def cleanhtml(raw_html, ctype):
@@ -375,11 +342,16 @@ def render_graph(result, cfg, **kwargs):
375342

376343
class CfgStruct:
377344
"""
378-
A simple class containing a bunch of fields
345+
A simple class containing a bunch of fields. Equivalent to Python3
346+
SimpleNamespace
379347
"""
380348
def __init__(self, **entries):
381349
self.__dict__.update(entries)
382350

351+
def __repr__(self):
352+
return '<' + ' '.join('{}={!r}'.format(*kv)
353+
for kv in self.__dict__.items()) + '>'
354+
383355

384356
# ----------------------------------------------------------------------
385357

@@ -394,198 +366,19 @@ def __init__(self, logger=None):
394366
self.log.info("START")
395367
self.cfg = CfgStruct(hdr=[], pfx={}, lmt=20, fmt=None, out=None, aut=None,
396368
grh=None, dis='table', typ=False, lan=[], par={},
397-
mth='GET', hhr=KeyCaseInsensitiveDict())
398-
399-
def magic(self, line):
400-
"""
401-
Read and process magics
402-
@param line (str): the full line containing a magic
403-
@return (list): a tuple (output-message,css-class), where
404-
the output message can be a single string or a list (containing
405-
a Python format string and its arguments)
406-
"""
407-
# The %lsmagic has no parameters
408-
if line.startswith('%lsmagic'):
409-
return magic_help, 'magic-help'
410-
411-
# Split line into command & parameters
412-
try:
413-
cmd, param = line.split(None, 1)
414-
except ValueError:
415-
raise KrnlException("invalid magic: {}", line)
416-
cmd = cmd[1:].lower()
417-
418-
# Process each magic
419-
if cmd == 'endpoint':
420-
421-
self.srv = SPARQLWrapper.SPARQLWrapper(param)
422-
return ['Endpoint set to: {}', param], 'magic'
423-
424-
elif cmd == 'auth':
425-
426-
auth_data = param.split(None, 2)
427-
if auth_data[0].lower() == 'none':
428-
self.cfg.aut = None
429-
return ['HTTP authentication: None'], 'magic'
430-
if auth_data and len(auth_data) != 3:
431-
raise KrnlException("invalid %auth magic")
432-
self.cfg.aut = auth_data
433-
return ['HTTP authentication: {}', auth_data], 'magic'
434-
435-
elif cmd == 'qparam':
436-
437-
v = param.split(None, 1)
438-
if len(v) == 0:
439-
raise KrnlException("missing %qparam name")
440-
elif len(v) == 1:
441-
self.cfg.par.pop(v[0], None)
442-
return ['Param deleted: {}', v[0]], 'magic'
443-
else:
444-
self.cfg.par[v[0]] = v[1]
445-
return ['Param set: {} = {}'] + v, 'magic'
446-
447-
elif cmd == 'http_header':
448-
449-
v = param.split(None, 1)
450-
if len(v) == 0:
451-
raise KrnlException("missing %http_header name")
452-
elif len(v) == 1:
453-
try:
454-
del self.cfg.hhr[v[0]]
455-
return ['HTTP header deleted: {}', v[0]], 'magic'
456-
except KeyError:
457-
return ['Not-existing HTTP header: {}', v[0]], 'magic'
458-
else:
459-
self.cfg.hhr[v[0]] = v[1]
460-
return ['HTTP header set: {} = {}'] + v, 'magic'
461-
462-
elif cmd == 'prefix':
463-
464-
v = param.split(None, 1)
465-
if len(v) == 0:
466-
raise KrnlException("missing %prefix value")
467-
elif len(v) == 1:
468-
self.cfg.pfx.pop(v[0], None)
469-
return ['Prefix deleted: {}', v[0]], 'magic'
470-
else:
471-
self.cfg.pfx[v[0]] = v[1]
472-
return ['Prefix set: {} = {}'] + v, 'magic'
473-
474-
elif cmd == 'show':
475-
476-
if param == 'all':
477-
self.cfg.lmt = None
478-
else:
479-
try:
480-
self.cfg.lmt = int(param)
481-
except ValueError as e:
482-
raise KrnlException("invalid result limit: {}", e)
483-
sz = self.cfg.lmt if self.cfg.lmt is not None else 'unlimited'
484-
return ['Result maximum size: {}', sz], 'magic'
485-
486-
elif cmd == 'format':
487-
488-
fmt_list = {'JSON': SPARQLWrapper.JSON,
489-
'N3': SPARQLWrapper.N3,
490-
'XML': SPARQLWrapper.XML,
491-
'RDF': SPARQLWrapper.RDF,
492-
'NONE': None,
493-
'DEFAULT': True,
494-
'ANY': False}
495-
try:
496-
fmt = param.upper()
497-
self.cfg.fmt = fmt_list[fmt]
498-
except KeyError:
499-
raise KrnlException('unsupported format: {}\nSupported formats are: {!s}', param, list(fmt_list.keys()))
500-
return ['Request format: {}', fmt], 'magic'
501-
502-
elif cmd == 'lang':
503-
504-
self.cfg.lan = DEFAULT_TEXT_LANG if param == 'default' else [] if param=='all' else param.split()
505-
return ['Label preferred languages: {}', self.cfg.lan], 'magic'
506-
507-
elif cmd in 'graph':
508-
509-
self.cfg.grh = param if param else None
510-
return ['Default graph: {}', param if param else 'None'], 'magic'
511-
512-
elif cmd == 'display':
513-
514-
v = param.lower().split(None, 2)
515-
if len(v) == 0 or v[0] not in ('table', 'raw', 'graph', 'diagram'):
516-
raise KrnlException('invalid %display command: {}', param)
517-
518-
msg_extra = ''
519-
if v[0] not in ('diagram', 'graph'):
520-
self.cfg.dis = v[0]
521-
self.cfg.typ = len(v) > 1 and v[1].startswith('withtype')
522-
if self.cfg.typ and self.cfg.dis == 'table':
523-
msg_extra = '\nShow Types: on'
524-
elif len(v) == 1: # graph format, defaults
525-
self.cfg.dis = ['svg']
526-
else: # graph format, with options
527-
if v[1] not in ('png', 'svg'):
528-
raise KrnlException('invalid graph format: {}', param)
529-
if len(v) > 2:
530-
if not v[2].startswith('withlit'):
531-
raise KrnlException('invalid graph option: {}', param)
532-
msg_extra = '\nShow literals: on'
533-
self.cfg.dis = v[1:3]
534-
535-
display = self.cfg.dis[0] if is_collection(self.cfg.dis) else self.cfg.dis
536-
return ['Display: {}{}', display, msg_extra], 'magic'
537-
538-
elif cmd == 'outfile':
539-
540-
if param in ('NONE', 'OFF'):
541-
self.cfg.out = None
542-
return ['no output file'], 'magic'
543-
else:
544-
self.cfg.out = param
545-
return ['Output file: {}', os.path.abspath(param)], 'magic'
546-
547-
elif cmd == 'log':
548-
549-
if not param:
550-
raise KrnlException('missing log level')
551-
try:
552-
lev = param.upper()
553-
parent_logger = logging.getLogger(__name__.rsplit('.', 1)[0])
554-
parent_logger.setLevel(lev)
555-
return ("Logging set to {}", lev), 'magic'
556-
except ValueError:
557-
raise KrnlException('unknown log level: {}', param)
558-
559-
elif cmd == 'header':
560-
561-
if param.upper() == 'OFF':
562-
num = len(self.cfg.hdr)
563-
self.cfg.hdr = []
564-
return ['All headers deleted ({})', num], 'magic'
565-
else:
566-
if param in self.cfg.hdr:
567-
return ['Header skipped (repeated)'], 'magic'
568-
self.cfg.hdr.append(param)
569-
return ['Header added: {}', param], 'magic'
570-
571-
elif cmd == 'method':
572-
573-
method = param.upper()
574-
if method not in ('GET', 'POST'):
575-
raise KrnlException('invalid HTTP method: {}', param)
576-
self.cfg.mth = method
577-
return ['HTTP method: {}', method], 'magic'
578-
579-
else:
580-
raise KrnlException("magic not found: {}", cmd)
369+
mth='GET', hhr=KeyCaseInsensitiveDict(), ept=None)
581370

582371

583372
def query(self, query, num=0, silent=False):
584373
"""
585374
Launch an SPARQL query, process & convert results and return them
586375
"""
587-
if self.srv is None:
376+
self.log.debug("CONFIG: %s", self.cfg)
377+
# Create server object, if needed
378+
if self.cfg.ept is None:
588379
raise KrnlException('no endpoint defined')
380+
elif self.srv is None or self.srv.endpoint != self.cfg.ept:
381+
self.srv = SPARQLWrapper.SPARQLWrapper(self.cfg.ept)
589382

590383
# Add to the query all predefined SPARQL prefixes
591384
if self.cfg.pfx:

0 commit comments

Comments
 (0)