Python多进程间共享对象 – Tenosr's notebook

开发并行程序的时候，会碰到在不同进程访问同一个对象的情况，python下面有个multiprocessing.Manager，使用这个类，可以创建共享对象(shared object)，其他进程就可以通过proxy访问这个这些共享对象了。

举个例子，定义一个共享的list对象，有两个写进程入，他们每秒在这个list写入”PID:进程ID, NUM:随机数”这样的内容，一个读取进程每2秒读取这个list的内容：

<span class="line"><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> multiprocessing</span></span>
<span class="line"><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> random</span></span>
<span class="line"><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> time</span></span>
<span class="line"><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> os</span></span>
<span class="line"></span>
<span class="line"><span style="color: #616E88"># 写入进程执行的内容</span></span>
<span class="line"><span style="color: #81A1C1">def</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">writer</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">my_list</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">    pid </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> os</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">getpid</span><span style="color: #ECEFF4">()</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">while</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">True</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">        time</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">sleep</span><span style="color: #ECEFF4">(</span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">)</span></span>
<span class="line"><span style="color: #D8DEE9FF">        my_list</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">append</span><span style="color: #ECEFF4">(</span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">PID:</span><span style="color: #EBCB8B">{}</span><span style="color: #A3BE8C">, NUM:</span><span style="color: #EBCB8B">{}</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">format</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">pid</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> random</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">randint</span><span style="color: #ECEFF4">(</span><span style="color: #B48EAD">0</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">100</span><span style="color: #ECEFF4">)))</span></span>
<span class="line"></span>
<span class="line"><span style="color: #616E88"># 读取进程执行的内容</span></span>
<span class="line"><span style="color: #81A1C1">def</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">reader</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">my_list</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">while</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">True</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">        time</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">sleep</span><span style="color: #ECEFF4">(</span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">)</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">len</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">my_list</span><span style="color: #ECEFF4">)</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">></span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">0</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">            </span><span style="color: #88C0D0">print</span><span style="color: #ECEFF4">(</span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">-----------------------------</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">)</span></span>
<span class="line"><span style="color: #D8DEE9FF">            </span><span style="color: #88C0D0">print</span><span style="color: #ECEFF4">(</span><span style="color: #ECEFF4">'</span><span style="color: #EBCB8B">\n</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">join</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">my_list</span><span style="color: #ECEFF4">))</span></span>
<span class="line"></span>
<span class="line"></span>
<span class="line"><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> __name__ </span><span style="color: #81A1C1">==</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">__main__</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">with</span><span style="color: #D8DEE9FF"> multiprocessing</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">Manager</span><span style="color: #ECEFF4">()</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">as</span><span style="color: #D8DEE9FF"> manager</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># 创建manager对象</span></span>
<span class="line"><span style="color: #D8DEE9FF">        my_list </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> manager</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">list</span><span style="color: #ECEFF4">()</span><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># 共享对象my_list</span></span>
<span class="line"><span style="color: #D8DEE9FF">        wp1 </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> multiprocessing</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">Process</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">target</span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF">writer</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">args</span><span style="color: #81A1C1">=</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">my_list</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">))</span></span>
<span class="line"><span style="color: #D8DEE9FF">        wp2 </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> multiprocessing</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">Process</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">target</span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF">writer</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">args</span><span style="color: #81A1C1">=</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">my_list</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">))</span></span>
<span class="line"><span style="color: #D8DEE9FF">        wp1</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">start</span><span style="color: #ECEFF4">()</span></span>
<span class="line"><span style="color: #D8DEE9FF">        wp2</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">start</span><span style="color: #ECEFF4">()</span></span>
<span class="line"></span>
<span class="line"><span style="color: #D8DEE9FF">        rp </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> multiprocessing</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">Process</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">target</span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF">reader</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">args</span><span style="color: #81A1C1">=</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">my_list</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">))</span></span>
<span class="line"><span style="color: #D8DEE9FF">        rp</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">start</span><span style="color: #ECEFF4">()</span></span>
<span class="line"></span>
<span class="line"><span style="color: #D8DEE9FF">        wp1</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">join</span><span style="color: #ECEFF4">()</span></span>
<span class="line"><span style="color: #D8DEE9FF">        wp2</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">join</span><span style="color: #ECEFF4">()</span></span>
<span class="line"><span style="color: #D8DEE9FF">        rp</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">join</span><span style="color: #ECEFF4">()</span></span>

代码比较简单，不废话了。

Manager定义在multiprocessing.managers.py文件下面，从文件里看，Manger()指向的其实是SyncManager

SyncManager继承自BaseManager，后面定义自己的manager还会用到BaseManager这个类。

SyncManager的内容:

<span class="line"><span style="color: #81A1C1">class</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">SyncManager</span><span style="color: #ECEFF4">(</span><span style="color: #8FBCBB; font-weight: bold">BaseManager</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #ECEFF4">'''</span></span>
<span class="line"><span style="color: #A3BE8C">    Subclass of `BaseManager` which supports a number of shared object types.</span></span>
<span class="line"></span>
<span class="line"><span style="color: #A3BE8C">    The types registered are those intended for the synchronization</span></span>
<span class="line"><span style="color: #A3BE8C">    of threads, plus `dict`, `list` and `Namespace`.</span></span>
<span class="line"></span>
<span class="line"><span style="color: #A3BE8C">    The `multiprocessing.Manager()` function creates started instances of</span></span>
<span class="line"><span style="color: #A3BE8C">    this class.</span></span>
<span class="line"><span style="color: #A3BE8C">    </span><span style="color: #ECEFF4">'''</span></span>

不是我懒，是它确实什么都没有～真的就只继承了BaseManager，SyncManager主要干的活是调用register来注册一些可以通过proxy访问的对象，比如上面我们用到的list对象，managers.py文件里是这么注册list对象的：

<span class="line"><span style="color: #D8DEE9FF">SyncManager</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">register</span><span style="color: #ECEFF4">(</span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">list</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">list</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> ListProxy</span><span style="color: #ECEFF4">)</span></span>

register是BaseManager的类方法，长这样：

<span class="line"><span style="color: #ECEFF4">@</span><span style="color: #D08770">classmethod</span></span>
<span class="line"><span style="color: #81A1C1">def</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">register</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">cls</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">typeid</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">callable</span><span style="color: #81A1C1">=None</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">proxytype</span><span style="color: #81A1C1">=None</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">exposed</span><span style="color: #81A1C1">=None</span><span style="color: #ECEFF4">,</span></span>
<span class="line"><span style="color: #D8DEE9FF">             </span><span style="color: #D8DEE9">method_to_typeid</span><span style="color: #81A1C1">=None</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">create_method</span><span style="color: #81A1C1">=True</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #ECEFF4">'''</span></span>
<span class="line"><span style="color: #A3BE8C">    Register a typeid with the manager type</span></span>
<span class="line"><span style="color: #A3BE8C">    </span><span style="color: #ECEFF4">'''</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">_registry</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">not</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">in</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">cls</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9FF">__dict__</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #81A1C1">cls</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9FF">_registry </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">cls</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9FF">_registry</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">copy</span><span style="color: #ECEFF4">()</span></span>
<span class="line"></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> proxytype </span><span style="color: #81A1C1">is</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">None</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">        proxytype </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> AutoProxy</span></span>

BaseManager类的_registry里面保存了所有注册过的类型，看下面：

如果proxytype是空的话，会调用AutoProxy。

上面扯了这么多，如果我们想共享一个自己的定义的类，怎么做？仿照SyncManger的写法就可以了，官方例子是这样的：

<span class="line"><span style="color: #81A1C1">from</span><span style="color: #D8DEE9FF"> multiprocessing</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9FF">managers </span><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> BaseManager</span></span>
<span class="line"></span>
<span class="line"><span style="color: #81A1C1">class</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">MathsClass</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">def</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">add</span><span style="color: #ECEFF4">(</span><span style="color: #81A1C1">self</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">x</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">y</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #81A1C1">return</span><span style="color: #D8DEE9FF"> x </span><span style="color: #81A1C1">+</span><span style="color: #D8DEE9FF"> y</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">def</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">mul</span><span style="color: #ECEFF4">(</span><span style="color: #81A1C1">self</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">x</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">y</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #81A1C1">return</span><span style="color: #D8DEE9FF"> x </span><span style="color: #81A1C1">*</span><span style="color: #D8DEE9FF"> y</span></span>
<span class="line"></span>
<span class="line"><span style="color: #81A1C1">class</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">MyManager</span><span style="color: #ECEFF4">(</span><span style="color: #8FBCBB; font-weight: bold">BaseManager</span><span style="color: #ECEFF4">):</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">pass</span></span>
<span class="line"></span>
<span class="line"><span style="color: #D8DEE9FF">MyManager</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">register</span><span style="color: #ECEFF4">(</span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">Maths</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> MathsClass</span><span style="color: #ECEFF4">)</span></span>
<span class="line"></span>
<span class="line"><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> __name__ </span><span style="color: #81A1C1">==</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">__main__</span><span style="color: #ECEFF4">'</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">    </span><span style="color: #81A1C1">with</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">MyManager</span><span style="color: #ECEFF4">()</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">as</span><span style="color: #D8DEE9FF"> manager</span><span style="color: #ECEFF4">:</span></span>
<span class="line"><span style="color: #D8DEE9FF">        maths </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> manager</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">Maths</span><span style="color: #ECEFF4">()</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #88C0D0">print</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">maths</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">add</span><span style="color: #ECEFF4">(</span><span style="color: #B48EAD">4</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">))</span><span style="color: #D8DEE9FF">         </span><span style="color: #616E88"># prints 7</span></span>
<span class="line"><span style="color: #D8DEE9FF">        </span><span style="color: #88C0D0">print</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">maths</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">mul</span><span style="color: #ECEFF4">(</span><span style="color: #B48EAD">7</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">8</span><span style="color: #ECEFF4">))</span><span style="color: #D8DEE9FF">         </span><span style="color: #616E88"># prints 56</span></span>

应该还是蛮简单的，如果想在自己的manager类里面注册list，可以仿照SyncManager的写法,MyManager.register(‘list’, list, ListProxy)。

Leave a Reply Cancel reply