Skip to content

Conversation

@zeedann
Copy link
Contributor

@zeedann zeedann commented Apr 10, 2017

Part of a rule on #36

simple-lambdas has been implemented.
This PR is for preference of Operator Fcns over Lambda Fcns (https://google.github.io/styleguide/pyguide.html?showone=Lambda_Functions#Lambda_Functions)

# if shopify_python.ast.count_tree_size(node) == 1:
# print node.name
if shopify_python.ast.count_tree_size(node) == 3:
operator = OperatorUtils().get_bin_op(node.op)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could instead write this as:

BINARY_OPERATORS = {
...
}
...
operator = BINARY_OPERATORS.get(node.op)
if operator:
    ...
    

And save yourself the getter functions and make those dicts constants.

Copy link
Contributor

@cfournie cfournie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

"-": "neg",
"+": "pos"
}
operator = unary_operators.get(node.body.op)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might as well check that this isn't None as well, e.g. if operator:

left = node.left.name
right = node.right.name
lamfun = "lambda " + left + ', ' + right + ": " + " ".join([left, node.op, right])
opfun = "Operator." + operator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operator should probably be lowercase since it's a module name.

def __lambda_func(self, node): # type: (astroid.Lambda) -> None
"""Prefer Operator Function to Lambda Functions"""

if isinstance(node.body, astroid.UnaryOp):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably want to make sure that the unary operation's operand is just a variable with the same id as the lambda's operand (e.g. you shouldn't recommend replacing lambda a: not (a + 3)

Copy link
Contributor Author

@zeedann zeedann Apr 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will ensuring that the unary operation's operand is not an instance of BinOp suffice?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not, check its type and make sure that it's the lambda variable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you will false-fail on f = lambda x: x * 3.

Please add a test to ensure that you don't.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erikwright just wondering, since we're preferring operator.mul(x, y) to f = lambda x, y: x * y, shouldn't we prefer operator.mul(x, 3) to f = lambda x: x * 3 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zeedann Here is the guideline you are implementing:

use the functions from the operator module instead of lambda functions

Here's the case we are discussing:

f = lambda x: x * 3

What is the example code that you are proposing?

f = lambda x: operator.mul(x, 3)

Why is this better? It is not replacing a lambda with anything. It's replacing a bare operator with a function call.

Here is an alternative that replaces the lambda:

f = functools.partial(operator.mul, 3)

But that's a bad idea :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I had a misinterpretation of what operator.mul actually did. Thanks for clarifying!

opfun = "Operator." + operator
self.add_message('lambda-func', node=node, args={'op': opfun, 'lamfun': lamfun})
return
elif shopify_python.ast.count_tree_size(node) > 3:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't want to be recommending partial replacement of lambdas, e.g. lambda a, b: a * b + c shouldn't be replaced with lambda a, b: operator.mul(a, b + c)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i be recommending lambda a, b, c; operator.mul(a, operator.add(b, c)) instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For common operations like multiplication, use the functions from the operator module instead of lambda functions. For example, prefer operator.mul to lambda x, y: x * y.

lambda a, b: a * b is definitely harder to read than operator.mul when used in reduce, e.g.

reduce(lambda a, b: a * b, integers)
reduce(operator.mul, integers)

But for larger expressions, I'd argue that lambda a, b, c: a * b + c is easier to read than lambda a, b, c: operator.mul(a, operator.add(b, c)), but it's debatable e.g.

c = 3
reduce(lambda a, b: a * b + c, integers)
reduce(lambda a, b: operator.mul(a, operator.add(b, c)), integers)

@erikwright @honkfestival @JasonMWhite thoughts?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. operator.sum over lambda a, b: a + b, but lambda a, b: (a * b) + b over operator.sum(operator.mul(a, b), b)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point guys, i'll recommend replacements for when lambda is used only on 2 variables with one operator

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of using operator is to not need the lambda. Not to never see a primitive operator.

@zeedann
Copy link
Contributor Author

zeedann commented Apr 11, 2017

UPDATE: changes I've made from the first iteration

  1. For the unary case, ensured that the operand is not a BinOp, and that it is the lambda variable.
  2. Removed the recursive code, to allow this to catch simpler lambda cases ( < 3 variables, 1 operation)

with self.assertNoMessages():
self.walk(root_not_msg)

def test_lambda_func(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also test a unary operator?
Also add a test that you don't reject lambda a, b: a * b * c

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split this test into a bunch of smaller tests, one for each case you are covering.

unary_root = astroid.builder.parse("""
def unaryfnc():
unary_pass = lambda x: not (x+3)
unary_msg = lambda x: -x
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a more realistic example, can you re-write these as some_var = map(operator.not_, iterable)

def binaryfnc():
binary_pass = lambda x, y, z: x * y + z
binary_pass2 = lambda x: x + 3
binary_msg = lambda x, y: x * y
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a more realistic example, can you re-write these as some_var = reduce(operator.mul, iterable)

@zeedann
Copy link
Contributor Author

zeedann commented Apr 12, 2017

@erikwright @JasonMWhite any final comments before I merge this? :D

opfun = "operator." + operator
self.add_message('lambda-func', node=node, args={'op': opfun, 'lamfun': lamfun})
elif isinstance(node.body, astroid.BinOp):
if shopify_python.ast.count_tree_size(node.body) == 3 and len(node.args.args) == 2:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't/shouldn't you put an equivalent check in the unary op case?

}
operator = unary_operators.get(node.body.op)
argname = node.args.args[0].name
if operator and not isinstance(node.body.operand, astroid.BinOp) and argname is node.body.operand.name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check for a BinOp operand seems fragile to me. Is that the only possible operand, besides a variable reference? Couldn't it be another unary operation? Or a function call?

IIUC the node type you actually expect is astroid.Name. Maybe look for that explicitly.

"not": "not_",
"+": "pos"
}
operator = unary_operators.get(node.body.op)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rename this to preferred_operator_function.

def binaryfnc():
binary_pass = reduce(lambda x, y, z: x * y + z, [1, 2, 3])
binary_pass2 = map(lambda x: x + 3, [1, 2, 3, 4])
binary_msg = reduce(lambda x, y: x * y, [1, 2, 3, 4])
Copy link
Contributor

@erikwright erikwright Apr 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add another test for:

binary_pass3 = reduce(lambda x, y: x + 1, [1, 2, 3])

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also for:

def binaryfnc():
    SOMETHING = 2
    ...
    binary_pass4 = reduce(lambda x, y: x * SOMETHING, [1,2,3,4])

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test for:

    binary_msg = reduce(lambda y, x: x * y, [1, 2, 3, 4])

}
operator = binary_operators.get(node.op)
if operator:
left = str(node.left.value) if node.left.name == 'int' else node.left.name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you verified that the operands were of the right node type you wouldn't have to worry about this case of literal operands, which I think you handle incorrectly anyway (see my requested test).

Copy link
Contributor Author

@zeedann zeedann Apr 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for catching that, i didn't handle it correctly. I've added a check to the if statement to check a match between the left & right arguments, and the left & right operands respectively

@zeedann
Copy link
Contributor Author

zeedann commented Apr 13, 2017

approval to merge? :D @erikwright @JasonMWhite


if isinstance(node.body, astroid.UnaryOp):
if shopify_python.ast.count_tree_size(node.body) == 2 and len(node.args.args) == 1:
unary_operators = {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of creating this variable on every node visitation, it would be better to create this as a class constant, since you aren't modifying it anywhere that I can see.

left_arg = node.args.args[0].name
right_arg = node.args.args[1].name
node = node.body
binary_operators = {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here with the constants.

if preferred_operator_function and left_arg is node.left.name and right_arg is node.right.name:
left = str(node.left.value) if node.left.name == 'int' else node.left.name
right = str(node.right.value) if node.right.name == 'int' else node.right.name
lamfun = "lambda " + left + ', ' + right + ": " + " ".join([left, node.op, right])

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor point, but lamfun doesn't seem very Pythonic as a name. lambda_fun? lamba_fx?

binary_msg = reduce(lambda x, y: x * y, [1, 2, 3, 4])
""")

# Binary Op Test 1: This will not trigger a message

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, these should be 4 separate tests.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, it could be one test, but you pass the whole thing in at once and assert that there is only one failure. Could be made a bit more clear by naming it binary_fail

with self.assertNoMessages():
self.walk(binary_pass2)

# Binary Op Test 3: This will trigger a message

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 2 Binary Op Test 3s

'op': 'operator.neg',
'lamfun': 'lambda x: - x'
})
with self.assertAddsMessages(unary_message):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However you decide to address the test comments for test_binary_lambda_func, do the same for test_unary_lambda_func

@zeedann zeedann force-pushed the lambda branch 2 times, most recently from 75e1dd1 to 6445c02 Compare April 17, 2017 17:38
"+": "pos"
}

binary_operators = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make these allcaps? (since they're constants)

@zeedann
Copy link
Contributor Author

zeedann commented Apr 17, 2017

I'm failing some tests that are unrelated to my PR.
I've tried updating/rebasing off master and use the make ** commands which succeed locally but fail on travis.
Here is an example: https://travis-ci.org/Shopify/shopify_python/jobs/222856408#L439
Officially stumped.
Any suggestions? :\

@JasonMWhite
Copy link

Can you reproduce these errors locally? Try clearing out your venv, and reinstalling, maybe a dependency got bumped?

@zeedann
Copy link
Contributor Author

zeedann commented Apr 17, 2017

can't reproduce these errors locally, everything looks fine locally

@cfournie
Copy link
Contributor

Probably not your fault, I just re-ran master and it failed, looks like they removed that function at some point from pylint (by no longer inheriting from the unit test class). We can probably downgrade pylint temporarily to fix.

upa

failed

new
@zeedann
Copy link
Contributor Author

zeedann commented Apr 17, 2017

Green has never looked so good 💚

Thanks @cfournie for the pylint update,

UPDATES:

  • moved dictionary of operators as constants of the class.
  • shortened tests to by removing all cases of assertNoMessages and assert that only one failure message will be added

@zeedann
Copy link
Contributor Author

zeedann commented Apr 20, 2017

Any last comments/thoughts before I merge this in the morning? :D

@zeedann zeedann merged commit d8dac26 into master Apr 21, 2017
@cfournie cfournie deleted the lambda branch April 21, 2017 14:08
@JasonMWhite JasonMWhite mentioned this pull request May 30, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants