Sharing Parameters between BucketingModules

[bricksdont]

Description

I am trying to share parameters between two BucketingModules. More specifically, my requirements are:

• I have different sources of data, that are read by different data iterators
• I have several instances of BucketingModule. The modules have some parameters in common, and they should share them. I.e. the parameter arrays should essentially be identical.
• On top of that, the modules have other parameters that they do not share.

What is your recommendation? How should I set up those modules?

Things I've tried

1. I tried defining a master module and then, while binding a second module, use the shared_module attribute:

https://mxnet.incubator.apache.org/api/python/module/module.html?highlight=bucketing#mxnet.module.BucketingModule.bind

Unfortunately, this is not available for BucketingModule. (Why?)

2. It's possible to get and set params between modules to sync some of their parameters after each update. Here is a working example:

https://gist.github.com/bricksdont/3dc6213482c38526decacf2a54b7f8b6

But the downside is obviously that it slows down training.

Environment info (Required)

$ python diagnose.py
----------Python Info----------
('Version      :', '2.7.14')
('Compiler     :', 'GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)')
('Build        :', ('default', 'Oct  5 2017 02:28:52'))
('Arch         :', ('64bit', ''))
------------Pip Info-----------
('Version      :', '9.0.1')
('Directory    :', '/Users/mathiasmuller/miniconda2/lib/python2.7/site-packages/pip')
----------MXNet Info-----------
/Users/mathiasmuller/miniconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/Users/mathiasmuller/miniconda2/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:46: DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom instead
  import OpenSSL.SSL
('Version      :', '1.0.0')
('Directory    :', '/Users/mathiasmuller/miniconda2/lib/python2.7/site-packages/mxnet')
('Commit Hash   :', '9ef196909ec7bf9cdda66d5b97c92793109798e1')
----------System Info----------
('Platform     :', 'Darwin-16.4.0-x86_64-i386-64bit')
('system       :', 'Darwin')
('node         :', 'dhcp-wlan-uzh-10-12-89-199.uzh.ch')
('release      :', '16.4.0')
('version      :', 'Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'i386')
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI
machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT FPU_CSDS
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.brand_string: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0053 sec, LOAD: 0.9015 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0222 sec, LOAD: 0.6658 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0220 sec, LOAD: 0.9553 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0020 sec, LOAD: 0.1104 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0199 sec, LOAD: 0.0656 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.9416 sec, LOAD: 0.0611 sec.

I'm using the Python package.

[bricksdont]

It seems that other people are having the very same problem: https://discuss.mxnet.io/t/sharing-parameters-between-two-modules-through-arg-dict/1046.

[chinakook]

You can hack module by yourself.

[bricksdont]

@chinakook Could you please elaborate on that? I assume you mean deriving a new class from BucketingModule, but what exactly do you recommend to do?

[chinakook]

you can refer to this hacking
initialize

mod = mx.mod.Module(sym, data_names=['data'], label_names=['softmax_label'], context=ctx)
mod.bind(data_shapes=[...]
         , label_shapes=[...])
mod.init_params(initializer=mx.init.Xavier(magnitude=2.))
mod.init_optimizer(optimizer='sgd', optimizer_params={'learning_rate':0.01, 'momentum': 0.9})

bind a new symbol

mod._symbol = new_sym
mod.binded=False
mod.bind(data_shapes=[...]
, label_shapes=[...], shared_module=mod)

You also need to hack parameters that the symbols does not share.

[bricksdont]

@chinakook Excuse me, I do not see how this solves my problem; BucketingModule does not have an argument shared_module. Binding the mod module only to rebind it later does not make much sense to me. And why exactly new_sym as the _symbol? As I understand, _symbol changes if the current bucket changes in a BucketingModule.

[ankkhedia]

@sandeep-krishnamurthy Could you please label it as question