Something that bothered me when I first started working with Python in Maya was reloading of modules. When working on tools that have multiple modules importing each other, it can be a pain to have to change something in one of the imported modules. There are a number of situations that arise where your modules just do not reload correctly. You might have imported an object from a module, or you might have imported a method with the same name as the module you imported it from, or you might have imported using an alias.
A situation came up last week at work that was somewhat related to this issue. Another one of the TDs was in the process of writing a daemon script, and we needed a way of ensuring that all of our modules get updated automatically without re-starting the daemon every single time we commit a change.
I came up with a quick solution before leaving for the weekend, but I felt there was a better solution available. I decided to take on a small home project over the weekend to come up with a more robust solution. With a bit of playing around I came up with a recursive solution utilizing the inspect module that seems to be working pretty well, so I thought I’d share :).
recursiveReload.py:
''' Created on 2012-02-25 @author: Mat ''' import sys import inspect from distutils import sysconfig from types import ModuleType _processedModules = set() _importedModules = set() _remappingData = list() class ImportRemapData(object): def __init__( self, destModuleName, destAttrName, sourceModuleName, sourceAttrName, attrType ): self._destModuleName = destModuleName self._destAttrName = destAttrName self._sourceModuleName = sourceModuleName self._sourceAttrName = sourceAttrName self._attrType = attrType def remap(self): ''' This gets the new modules and assigns the new attr value to the attr name in the new destination module. ''' #get the newly reloaded source module and then the new attr value sourceModule = sys.modules[self._sourceModuleName] #we need to get the new attr value as well newAttrVal = None #if the new attr val is a module, we need to get the module from sys if self._attrType == ModuleType: newAttrVal = sys.modules[self._sourceAttrName] #otherwise, just get it from the new source module else: newAttrVal = getattr(sourceModule, self._sourceAttrName) #get the newly reloaded destination module and set the new attr value destModule = sys.modules[self._destModuleName] setattr(destModule, self._destAttrName, newAttrVal) def recursiveReload(module, debug=False): #get all of the data necessary for full reload of the module _getModuleData(module) #reload all of the collected modules for importedModule in _importedModules: if debug: print 'Reloading module: %s' % importedModule reload(importedModule) #now that the modules are all reloaded, re-map the extra data for remapData in _remappingData: remapData.remap() #clear our variables for future use _processedModules.clear() _importedModules.clear() del _remappingData[:] return True def _getModuleData(module): ''' Inspects the module and all of its sub-modules to get all imported modules and data required for us to re-map variables properly. ''' #add this module to the list of processed modules so we do not try to #process it again and risk getting into an infinite loop _processedModules.add(module) #get all attr names from the module attrNames = dir(module) detectedModules = set() for attrName in attrNames: #get attr value and try to get the module where the value originated from attrVal = getattr(module, attrName) attrModule = inspect.getmodule(attrVal) #if we didn't get a module returned, we are dealing with a builtin, or #we are dealing with this module skip to the next loop iteration. #We need to do two checks for builtin modules, if it is in the builtin #module names and if it does not have a __file__ attr. if not attrModule \ or attrModule.__name__ in sys.builtin_module_names \ or not hasattr(attrModule, '__file__') \ or attrModule == sys.modules[__name__]: continue #we also do not want to reload if the module comes from the standard lib #to check for this, we need to get the module's path and check that it #is in the lib folder, but not in the site packages folder moduleFilepath = attrModule.__file__.lower() pyStdLib = sysconfig.get_python_lib(standard_lib=True).lower() pySitePkg = sysconfig.get_python_lib().lower() if moduleFilepath.startswith(pyStdLib) \ and not moduleFilepath.startswith(pySitePkg): continue #add the module to the list of modules that have been imported detectedModules.add(attrModule) #if the attr value does not have a name attr, skip to the next iteration if not hasattr(attrVal, '__name__'): continue #if the module where the attr value came from is different from the #current module, collect data we need to remap things origName = attrVal.__name__ if attrModule != module: #print module.__name__, attrName, attrModule.__name__, origName remapData = ImportRemapData( module.__name__, attrName, attrModule.__name__, origName, type(attrVal) ) _remappingData.append(remapData) #add the detected modules to the imported modules set _importedModules.update(detectedModules) #cycle through the modules we detected, which have not already been processed #and process them as well for detectedModule in detectedModules: if detectedModule in _processedModules: continue _getModuleData(detectedModule) return True |
The recursiveReload method within the module is what is of interest here. To use this script, simply pass a module to this method. It will collect all modules that have been imported into the current working module, as well as all classes and modules that have been imported from modules or imported using aliases. It will then search each of the collected sub modules for the same information, and so on and so forth. When it hits the bottom, it will finish searching and then begin the reloading process. It will reload all of the modules, and then re-map extra data such as methods and classes and aliases.
Here is an example of the script in action. Save the following sample scripts in the same directory as the recursiveReload.py module seen above. Run test1a.py and then go and edit and save the test2.py and test3.py files while test1a.py is still running. Not everything will properly reload. Now, do the same with test1b.py. All changes should reload.
test1a.py:
import test2 from time import sleep while True: print 'This is test1a.py' reload(test2) test2.printStatement() sleep(5) |
test1b.py:
import sys import test2 from time import sleep from recursiveReload import recursiveReload while True: print 'This is test1b.py' thisModule = sys.modules[__name__] recursiveReload(thisModule, debug=True) test2.printStatement() sleep(5) |
test2.py:
from test3 import anotherPrintStatement from test3 import PrintMultiplication as PrintMulti def printStatement(): print 'This is test2.py' anotherPrintStatement() PrintMulti(1, 2) |
test3.py:
def anotherPrintStatement(): print 'This is test3.py' class PrintMultiplication(object): def __init__(self, x, y): print x * y |
Feel free to continue adding more modules and make the importation chain even longer. It should continue to work.
You can pass True to the optional argument debug to have information printed to the console about which modules are being reloaded. The method will not reload builtin modules, the recursiveReload module, the __main__ module (it qualifies as a builtin anyways), and any modules that originate from the Python installation’s standard library but not in the site packages.
There are a few situations where this module currently will not work. It will fail when importing variables from other modules that are storing basic data types such as strings, ints, floats, booleans. They cannot be traced back to the module where they originated from using the inspect.getmodule method.
This will also fail if you change an import statement from from moduleName import moduleName to import moduleName, if moduleName is the name of the module and then name of an object from within the module that you are importing. I have yet to get this change properly remapping. You can, however, change an import statement from from moduleName import methodName to import moduleName with no issues.