How to dump objects of a running Python program?


1 weeks ago


25 time


I have a Python program which basically keeps list of Counter objects and then writes them to disk. After four days counting has finished but the system is almost out of 63GB memory and it has already swapped out 50GB and no progress.

Here is a simplified version of my code.

import os
import time
from collections import Counter


counters = [Counter() for i in range(4)]

while True:
    for i in range(1024):
        for counter in counters:
            counter[i] = 1
    with open('/tmp/counter.txt', 'w') as f:
        for counter in counters:
            f.write('\n'.join(map(str, counter.most_common())))

I am guessing it is stuck at the last line and cannot sort the dict because it is OOM.

I need to safely write these Counter objects to disk for later processing

In other threads I found some answers but couldn't work it out. Here is what I tried so far:

  1. Attach gdb to my Python program: gdb python3 32610
  2. Show backtrace: bt
  3. Try to pickle.dump possible candidate for Counter object

    (gdb) bt
    #21 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f70a45d0cf8, for file /usr/lib/python3.6/collections/, line 553, in most_common (self=<Counter at remote 0x7f70a2d7be60>, n=1048575)) at ../Python/ceval.c:4166
    (gdb) python i = gdb.inferiors()[0]
    (gdb) python m = i.read_memory(0x7f70a45d0cf8, 4)
    (gdb) python print(m.tobytes())
    (gdb) python import pickle
    (gdb) python pickle.dump(m, open('/tmp/02.pickle', 'wb'))
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    TypeError: can't pickle memoryview objects
    Error while executing Python code.
    (gdb) dump value 0x7f70a45d0cf8
    No value to dump.
  4. I am not sure how to find the start/end address of the objects I am interested in.

  5. Counter is not always shown in the backtrace. How I select the correct frame?
  6. Pickle won't pickle? I cannot install any of suggested cPickle or Garlicsim or meliae

0 answers