%run does not work with import hooks #11736

Activity

1. drunkwcodes commented on May 17, 2019

   

   drunkwcodes

   on May 17, 2019

   Contributor

   More actions

   Pull request?
2. Carreau commented on May 18, 2019

   

   Carreau

   on May 18, 2019

   Member

   More actions

   This is a really vague bug report. We almost never use imp (it is deprecated) , though we do use importlib and __import__ in many places. It will be hard to respond without narrowing what you wish / what your actual bug is.
3. mlucool commented on May 20, 2019

   

   mlucool

   on May 20, 2019

   Author

   More actions

   Sorry that I was not clear. Here is a reproducer:
   IPython: 7.3.0
   Python: 3.7.3

   Code:

   FULLPATH = '/tmp/my_tmp.py'
   with open(FULLPATH, 'w') as f:
       f.write('''
   print('Loaded my_tmp')
   if __name__ == "__main__":
       print('I just ran a script')
   ''')
   class MyTempImporter(object):
       def __init__(self):
           pass
    
       def find_module(self, fullname, path=None):
           if 'my_tmp' in fullname:
               return self
           return None
    
       def load_module(self, name):
           import imp
           return imp.load_source('my_tmp', FULLPATH)
       
       def get_code(self, fullname):
           import imp
           with open(FULLPATH, 'r') as f:
               compile(f.read(), 'foo', 'exec')
       def is_package(self, __):
           return False
    
   import sys
   sys.meta_path.insert(0, MyTempImporter())
   

   In [2]:  %run -m my_tmp                                                                                                                                                                        
   /usr/local/python/python-3.7/latest/lib64/python3.7/site-packages/IPython/core/magics/execution.py:683: UserWarning: 'my_tmp' is not a valid modulename on sys.path
     warn('%r is not a valid modulename on sys.path'%modulename)
   
   In [3]: import my_tmp                                                                                                                                                                          
   /usr/local/python/python-3.7/latest/bin/ipython:18: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
   Loaded my_tmp
   

   $ PYTHONPATH=/tmp python -m my_tmp
   Loaded my_tmp
   I just ran a script
   

   I have opened this to fix the issue in In[2]. Please let me know if more info will help.
4. Carreau commented on May 21, 2019

   

   Carreau

   on May 21, 2019

   Member

   More actions

   I'm confused; in the above code you are using imp, not us; so I'm not sure what you are asking about imp:

   $ rg 'from imp '
   

   IPython/extensions/autoreload.py
   120:from imp import reload
   
   IPython/utils/tests/test_path.py
   15:from imp import reload
   

    ~/dev/ipython[master ✗] $ rg 'import imp$'
   IPython/lib/deepreload.py
   37:import imp
   

   And those are extensions you do not seem to use, so not sure why imp would be involved in our codebase.

   As for run -m, it "simply" calls runpy.run_module, we may need to have some special handling for import hook. We can't use __import__ as we don't import it, we run in a given namespace, and properly set some things like whether it is __main__ or not.

   I would have to re-read about importlib to understand how to fix things.
5. mlucool commented on May 21, 2019

   

   mlucool

   on May 21, 2019

   Author

   More actions

   I'm confused; in the above code you are using imp, not us; so I'm not sure what you are asking about imp:

   I copied some code, I didn't mean to imply that using imp is the right way to build this hook. Only to show how to make the error. Sorry for the additional confusion.

   As for run -m, it "simply" calls runpy.run_module, we may need to have some special handling for import hook.

   Thanks! It looks like I wrongly assumed the bug's cause. Given something similar to the above reproducer, using %run is what I am after having fixed. Please let me know if you need any more information. I have updated the title.
6. mlucool

   changed the title [-]imp vs __import__[/-] [+]%run does not work with import hooks[/+] on May 21, 2019
7. ngoldbaum commented on Mar 4, 2020

   

   ngoldbaum

   on Mar 4, 2020

   Last edited by ngoldbaum

   Contributor

   More actions

   @mlucool when I tried to reproduce this just now using your example, I get:

   $ PYTHONPATH=/tmp ipython
   Python 3.7.3 (default, May 24 2019, 16:15:07)
   Type 'copyright', 'credits' or 'license' for more information
   IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.
   
   In [1]: %run -m my_tmp
   Loaded my_tmp
   I just ran a script
   
   In [2]: import my_tmp
   Loaded my_tmp
   

   This seems correct to me - is there something else that should be happening?
8. ngoldbaum commented on Mar 4, 2020

   

   ngoldbaum

   on Mar 4, 2020

   Last edited by ngoldbaum

   Contributor

   More actions

   Ah, I see, I guess this is the real reproducer:

   $ ipython
   Python 3.7.3 (default, May 24 2019, 16:15:07)
   Type 'copyright', 'credits' or 'license' for more information
   IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.
   
   In [1]: %run test.py # test.py is the code in https://github.com/ipython/ipython/issues/11736#issuecomment-494168069
   
   In [2]: %run -m my_tmp
   /home/goldbaum/Documents/ipython/IPython/core/magics/execution.py:675: UserWarning: 'my_tmp' is not a valid modulename on sys.path
     warn('%r is not a valid modulename on sys.path'%modulename)
   
   In [3]: import my_tmp
   Loaded my_tmp
   
9. Carreau

   closed this as completedin #12171on Mar 14, 2020