Skip to content

Commit 1180fc1

Browse files
georgeRobertsonmurd0
georgeRobertson
authored and
murd0
committed
Output checker (#96)
* First commit. * Modify autopep8 indents for readability. * Expand docstring example to all parameters. * Changed return to use any() * Mover any() to wrap set comaprison iterator * any() was incorrect, all() works. * null_count function * Add detail. * Made changes to check_null by removing unnecessary code and added fix to examples * Fixed examples * Fixed examples * Fixed examples * Rewording for clarity * Update CONTRIBUTING.md * null_count modified to output an integer * fixed example issues * First * check_measures_test added * Checks for value and key errors. Also, columns_to_be_checked changed to only accept tuple. * missing value errors * Fixed null_checker tests to find value errors and key errors * Fixed null checker for tests * `any` function to `any` method * fix missing value error * remove george's local settings * Added checknatval function * fixed issue on line 79 with \ * corrected missing bracket line 88 * Fixed example * regularised filenames * Initial commit. * Typo fix * Use python 3.6 * Scale back testing * added check_consistent_submissions_test * fixed submissions_test
1 parent 96f12ce commit 1180fc1

10 files changed

Lines changed: 579 additions & 12 deletions

‎CONTRIBUTING.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,25 @@
11
# How to contribute
22

3-
First off, thank you for taking the time to contribute!
3+
First off, thank you for taking the time to contribute! If you have a functionality that you would like to see in codon, we have a few standards and guidelines so we can merge your pull request quicker.
44

55
## Basic idea
66

7-
Source + Doc + Test = :heart_eyes:
7+
1. [Fork](https://help.github.com/en/articles/fork-a-repo) codonPython on GitHub.
88

9-
Please send a [GitHub Pull Request to codonPython](https://github.com/codonlibrary/codonPython/pull/new/master) with a
10-
clear description of what you have done.
11-
We suggest you follow our coding conventions (below) and make sure all of your commits are atomic (one feature per commit).
12-
Please do not send us undocumented code as we might not accept it. Including tests to your pull request brings tears of joy to our eyes.
9+
2. Write your documented function and tests (:heart_eyes:) on a new branch, coding in line with our **coding conventions**.
10+
11+
3. Submit a [pull request](https://help.github.com/en/articles/creating-a-pull-request) to codonPython with a clear description of what you have done.
12+
13+
We suggest you make sure all of your commits are atomic (one feature per commit). Please make sure that non-obvious lines of code are commented, and variable names are as clear as possible. Please do not send us undocumented code as we will not accept it. Including tests to your pull request will bring tears of joy to our eyes, and will also probably result in a faster merge.
1314

14-
The easier it is for us to review a pull request, the faster we will merge it.
1515

1616
## Coding conventions
1717

18-
Start reading our code and you will get an idea of it:
18+
Start reading our code to get a feel for it:
1919

20-
* We use [PEP8](https://www.python.org/dev/peps/pep-0008/).
20+
* We use [PEP8](https://www.python.org/dev/peps/pep-0008/). Autoformatters for PEP8, for instance [autopep8](https://pypi.org/project/autopep8/), can easily ensure compliance.
2121
* We use docstrings and we try to (loosely) follow [`numpy`'s docstring standards](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard).
2222
* This is open source software. Consider the people who will read your code, and make it look nice for them.
2323

24-
Thank you,
25-
codonPeople
24+
:clinking_glasses: Thank you!
25+
Team codon

‎codonPython/check_consistent_measures.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
import pandas as pd
2+
import numpy as np
3+
4+
5+
def check_consistent_measures(data, geography_col: str = "Org_Level", measure_col: str = "Measure", measures_set: set = set()) -> bool:
6+
"""
7+
Check every measure is in every geography level.
8+
9+
Parameters
10+
----------
11+
data : pd.DataFrame
12+
DataFrame of data to check.
13+
geography_col : str, default = "Org_Level"
14+
Column name for the geography level.
15+
measure_col : str, default = "Measure"
16+
Column name for measure
17+
measures_set : set, default = set()
18+
Set of measures that should be in every geography level. If empty, the existing
19+
global set is presumed to be correct.
20+
21+
Returns
22+
-------
23+
bool
24+
Whether the checks have been passed.
25+
26+
Examples
27+
--------
28+
>>> check_consistent_measures(
29+
... pd.DataFrame({
30+
... "Geog" : ["National" ,"National", "Region", "Region", "Local", "Local",],
31+
... "measure" : ["m1", "m2", "m1", "m2", "m1", "m2",],
32+
... "Value_Unsuppressed" : [4, 2, 2, 1, 2, 1,],
33+
... }),
34+
... geography_col = "Geog",
35+
... measure_col = "measure",
36+
... measures_set = set({"m1", "m2"}),
37+
... )
38+
True
39+
>>> check_consistent_measures(
40+
... pd.DataFrame({
41+
... "Org_Level" : ["National" ,"National", "Region", "Region", "Local", "Local",],
42+
... "Measure" : ["m1", "m3", "m1", "m2", "m1", "m2",],
43+
... "Value_Unsuppressed" : [4, 2, 2, 1, 2, 1,],
44+
... })
45+
... )
46+
False
47+
"""
48+
49+
if data.isna().any(axis=None):
50+
raise ValueError(
51+
f"Missing values at locations {list(map(tuple, np.argwhere(data.isna().values)))}"
52+
)
53+
if not isinstance(geography_col, str) or not isinstance(measure_col, str):
54+
raise ValueError("Please input strings for column indexes.")
55+
if not isinstance(measures_set, set):
56+
raise ValueError("Please input a set object for measures")
57+
if geography_col not in data.columns or measure_col not in data.columns:
58+
raise KeyError("Check column names correspond to the DataFrame.")
59+
60+
# Every geography level should have the same set of measures as the global set.
61+
global_set = measures_set if measures_set else set(data[measure_col].unique())
62+
subsets = data.groupby(geography_col) \
63+
.agg({measure_col: "unique"})
64+
subset_agreement = all(set(x) == global_set for x in subsets[measure_col])
65+
66+
return subset_agreement

‎codonPython/check_consistent_submissions.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
import pandas as pd
2+
3+
4+
def check_consistent_submissions(data, national_geog_level: str = "National", geography_col: str = "Org_Level", submissions_col: str = "Value_Unsuppressed", measure_col: str = "Measure", ) -> bool:
5+
"""
6+
Check total submissions for each measure are the same across all geography levels
7+
except national.
8+
9+
Parameters
10+
----------
11+
data : pd.DataFrame
12+
DataFrame of data to check.
13+
national_geog_level : str, default = "National"
14+
Geography level code for national values.
15+
geography_col : str, default = "Org_Level"
16+
Column name for the geography level.
17+
submissions_col : str, default = "Value_Unsuppressed"
18+
Column name for the submissions count.
19+
measure_col : str, default = "Measure"
20+
Column name for measure.
21+
22+
Returns
23+
-------
24+
bool
25+
Whether the checks have been passed.
26+
27+
Examples
28+
--------
29+
>>> check_consistent_submissions(
30+
... pd.DataFrame({
31+
... "Geog" : ["N" ,"N", "Region", "Region", "Local", "Local",],
32+
... "measure" : ["m1", "m2", "m1", "m2", "m1", "m2",],
33+
... "submissions" : [4, 2, 2, 1, 2, 1,],
34+
... }),
35+
... national_geog_level = "N",
36+
... geography_col = "Geog",
37+
... submissions_col = "submissions",
38+
... measure_col = "measure",
39+
... )
40+
True
41+
>>> check_consistent_submissions(
42+
... pd.DataFrame({
43+
... "Org_Level" : ["National" ,"National", "Region", "Region", "Local", "Local",],
44+
... "Measure" : ["m1", "m2", "m1", "m2", "m1", "m2",],
45+
... "Value_Unsuppressed" : [4, 2, 3, 1, 2, 1,],
46+
... })
47+
... )
48+
False
49+
"""
50+
51+
if (
52+
not isinstance(submissions_col, str) or
53+
not isinstance(measure_col, str) or
54+
not isinstance(geography_col, str) or
55+
not isinstance(national_geog_level, str)
56+
):
57+
raise ValueError("Please input strings for column names and national geography level.")
58+
if (
59+
submissions_col not in data.columns or
60+
measure_col not in data.columns or
61+
geography_col not in data.columns
62+
):
63+
raise KeyError("Check column names correspond to the DataFrame.")
64+
65+
# All non-national measures should have only one unique submission number for each
66+
# geography level.
67+
submissions_by_measure = data[data[geography_col] != national_geog_level] \
68+
.groupby(measure_col) \
69+
.agg({submissions_col: "nunique"})
70+
result = (submissions_by_measure[submissions_col] == 1).all()
71+
72+
return result

‎codonPython/check_nat_val.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
import pandas as pd
2+
3+
4+
def check_nat_val(df: pd.DataFrame, breakdown_col: str = "Breakdown",
5+
measure_col: str = "Measure", value_col: str =
6+
"Value_Unsuppressed", nat_val: str = "National") -> bool:
7+
"""
8+
Check national value less than or equal to sum of breakdowns.
9+
10+
This function checks that the national value is less than or equal to the
11+
sum of each organisation level breakdown.
12+
This function does not apply to values which are averages.
13+
This function does not apply to values which are percentages calculated
14+
from the numerator and denominator.
15+
16+
Parameters
17+
----------
18+
df : pandas.DataFrame
19+
DataFrame of data to check.
20+
breakdown_col : str, default = "Breakdown"
21+
Column name for the breakdown level.
22+
measure_col : str, default = "Measure"
23+
Column name for measures
24+
value_col : str, default = "Value_Unsuppressed"
25+
Column name for values
26+
nat_val : str, default = "National"
27+
Value in breakdown column denoting national values
28+
Returns
29+
-------
30+
bool
31+
Whether the checks have been passed.
32+
33+
Examples
34+
--------
35+
>>> check_nat_val(
36+
... df = pd.DataFrame({
37+
... "Breakdown" : ['National', 'CCG', 'CCG', 'Provider', 'Provider',
38+
... 'National' ,'CCG', 'CCG', 'Provider', 'Provider','National' ,'CCG', 'CCG',
39+
... 'Provider', 'Provider',],
40+
... "Measure" : ['m1', 'm1', 'm1', 'm1', 'm1', 'm2', 'm2', 'm2', 'm2',
41+
... 'm2', 'm3', 'm3', 'm3', 'm3', 'm3',],
42+
... "Value_Unsuppressed" : [9, 4, 5, 3, 6, 11, 2, 9, 7, 4, 9, 5, 4, 6,
43+
... 3],
44+
... }),
45+
... breakdown_col = "Breakdown",
46+
... measure_col = "Measure",
47+
... value_col = "Value_Unsuppressed",
48+
... nat_val = "National",
49+
... )
50+
True
51+
>>> check_nat_val(
52+
... df = pd.DataFrame({
53+
... "Breakdown" : ['National', 'CCG', 'CCG', 'Provider', 'Provider',
54+
... 'National' ,'CCG', 'CCG', 'Provider', 'Provider','National' ,'CCG', 'CCG',
55+
... 'Provider', 'Provider',],
56+
... "Measure" : ['m1', 'm1', 'm1', 'm1', 'm1', 'm2', 'm2', 'm2', 'm2',
57+
... 'm2', 'm3', 'm3', 'm3', 'm3', 'm3',],
58+
... "Value_Unsuppressed" : [18, 4, 5, 3, 6, 11, 2, 9, 7, 4, 9, 5, 4, 6,
59+
... 3],
60+
... }),
61+
... breakdown_col = "Breakdown",
62+
... measure_col = "Measure",
63+
... value_col = "Value_Unsuppressed",
64+
... nat_val = "National",
65+
... )
66+
False
67+
"""
68+
69+
if not isinstance(breakdown_col, str) or not isinstance(measure_col, str)\
70+
or not isinstance(value_col, str):
71+
raise ValueError("Please input strings for column indexes.")
72+
if not isinstance(nat_val, str):
73+
raise ValueError("Please input strings for value indexes.")
74+
if breakdown_col not in df.columns or measure_col not in df.columns or\
75+
value_col not in df.columns:
76+
raise KeyError("Check column names correspond to the DataFrame.")
77+
# aggregate values by measure and breakdown
78+
grouped = df.groupby([measure_col, breakdown_col]).agg({value_col: sum})\
79+
.reset_index()
80+
national = grouped.loc[grouped[breakdown_col] == nat_val].reset_index()
81+
non_national = grouped.loc[grouped[breakdown_col] != nat_val].reset_index()
82+
# check values are less than or equal to national value for each measure
83+
join = pd.merge(non_national, national, left_on=measure_col,
84+
right_on=measure_col, how='left')
85+
left = value_col + '_x'
86+
right = value_col + '_y'
87+
join['Check'] = join[right] <= join[left]
88+
result = all(join['Check'])
89+
return result

‎codonPython/check_null.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import numpy
2+
import pandas as pd
3+
4+
def check_null(dataframe: pd.DataFrame, columns_to_be_checked: list) -> bool:
5+
"""
6+
Checks a pandas dataframe for null values
7+
8+
This function takes a pandas dataframe supplied as an argument and returns a integer value representing any null values found within the columns to check
9+
10+
Parameters
11+
----------
12+
data : pandas.DataFrame
13+
Dataframe to read
14+
columns_to_be_checked: list
15+
Given dataframe columns to be checked for null values
16+
17+
Returns
18+
-------
19+
out : int
20+
The number of null values found in the given columns
21+
22+
Examples
23+
--------
24+
>>> check_null(dataframe = pd.DataFrame({'col1': [1,2], 'col2': [3,4]}),columns_to_be_checked = ['col1', 'col2'])
25+
0
26+
>>> check_null(dataframe = pd.DataFrame({'col1': [1,numpy.nan], 'col2': [3,4]}),columns_to_be_checked = ['col1'])
27+
1
28+
"""
29+
30+
if not isinstance(columns_to_be_checked, list):
31+
raise ValueError("Please make sure that all your columns passed are strings")
32+
else:
33+
pass
34+
35+
for eachCol in columns_to_be_checked:
36+
if eachCol not in dataframe.columns:
37+
raise KeyError("Please check the column names correspond to values in the DataFrame.")
38+
else:
39+
pass
40+
41+
null_count = 0
42+
for eachColumn in columns_to_be_checked:
43+
prev_null_count = null_count
44+
null_count = prev_null_count + (len(dataframe) - dataframe[eachColumn].count())
45+
46+
return null_count

‎codonPython/nhsNumber.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ def nhsNumberValidator(number: int) -> bool:
5151

5252
def nhsNumberGenerator(to_generate: int, random_state: int = None) -> list:
5353
"""
54-
Generates up to 1M random NHS number(s) compliant with modulus 11 checks recorded
54+
Generates up to 1M random NHS numbers compliant with modulus 11 checks as recorded
5555
in the data dictonary.
5656
https://www.datadictionary.nhs.uk/data_dictionary/attributes/n/nhs/nhs_number_de.asp?shownav=1
5757

‎codonPython/tests/check_consistent_measures_test.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
from codonPython.check_consistent_measures import check_consistent_measures
2+
import pandas as pd
3+
import numpy as np
4+
import pytest
5+
6+
@pytest.mark.parametrize("data, geography_col, measure_col, measures_set, expected", [
7+
(
8+
pd.DataFrame({
9+
"Geog" : ["National" ,"National", "Region", "Region", "Local", "Local"],
10+
"measure" : ["m1", "m2", "m1", "m2", "m1", "m2"],
11+
"Value_Unsuppressed" : [4, 2, 2, 1, 2, 1],
12+
}),
13+
"Geog",
14+
"measure",
15+
set({"m1", "m2"}),
16+
True
17+
),
18+
(
19+
pd.DataFrame({
20+
"Geog" : ["National" ,"National", "Region", "Region", "Local", "Local"],
21+
"measure" : ["m1", "m2", "m1", "m3", "m1", "m2"],
22+
"Value_Unsuppressed" : [4, 2, 2, 1, 2, 1],
23+
}),
24+
"Geog",
25+
"measure",
26+
set({"m1", "m2"}),
27+
False
28+
)
29+
])
30+
31+
def test_each_org_levels_BAU(data, geography_col, measure_col, measures_set, expected):
32+
assert expected == check_consistent_measures(data, geography_col, measure_col, measures_set)
33+
34+
35+
@pytest.mark.parametrize("data, geography_col, measure_col, measures_set", [
36+
(
37+
pd.DataFrame({
38+
"Geog" : ["National" ,"National", "Region", "Region", "Local", "Local"],
39+
"measure" : ["m1", "m2", "m1", np.nan, "m1", "m2"],
40+
"Value_Unsuppressed" : [4, 2, 2, 1, 2, 1],
41+
}),
42+
"Geog",
43+
"measure",
44+
set({"m1", "m2"}),
45+
),
46+
])
47+
48+
49+
def test_each_org_levels_valueErrors_measure_col(data, geography_col, measure_col, measures_set):
50+
with pytest.raises(ValueError):
51+
check_consistent_measures(data, geography_col, measure_col, measures_set)
52+
53+
@pytest.mark.parametrize("data, geography_col, measure_col, measures_set", [
54+
(
55+
pd.DataFrame({
56+
"Geog" : ["National" ,"National", "Region", "Region", "Local", "Local"],
57+
"measure" : ["m1", "m2", "m1", "m2", "m1", "m2"],
58+
"Value_Unsuppressed" : [4, 2, 2, 1, 2, 1],
59+
}),
60+
"Global",
61+
"measure",
62+
set({"m1", "m2"}),
63+
)
64+
])
65+
66+
def test_each_geography_col_keyError(data, geography_col, measure_col, measures_set):
67+
with pytest.raises(KeyError):
68+
check_consistent_measures(data, geography_col, measure_col, measures_set)

0 commit comments

Comments
 (0)