Questions tagged [multiprocessing]

1

votes
4

answer
974

Views

Python: How to run nested parallel process in python?

I have a dataset df of trader transactions. I have 2 levels of for loops as follows: smartTrader =[] for asset in range(len(Assets)): df = df[df['Assets'] == asset] # I have some more calculations here for trader in range(len(df['TraderID'])): # I have some calculations here, If trader is successful...
roland
1

votes
0

answer
18

Views

multiprocessing in python does not stop running

I tried a simple example of multiprocessing in python from their website itself, but it does not give any input. It's showing as running itself and I am not able to stop it in jupyter notebook. from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': p = Pool(5) print(p.map(f...
Sachin Sudheer
1

votes
1

answer
254

Views

Python multiprocess Queue get() blocks

I wrote a tiny proxy utility (tcp port-forwarder), and now trying to test it. So my test: def test_forwarding(self): route = self.config.routes[0] q = multiprocessing.JoinableQueue() proc_listen = multiprocessing.Process( target=lambda q: q.put(str( subprocess.Popen(["nc", "-l", str(route.sink[1])],...
Constantine
1

votes
2

answer
229

Views

How to get `setup.py test` working with `multiprocessing` on Windows?

Issue I am trying to get setup.py test working with a package that uses multiprocessing on Windows. I have the following scenario: A regular python package folder structure with a package a test folder and a setup.py. Running python -m unittest from the top directory executes my tests without comp...
NOhs
12

votes
3

answer
245

Views

How to efficiently use asyncio when calling a method on a BaseProxy?

I'm working on an application that uses LevelDB and that uses multiple long-lived processes for different tasks. Since LevelDB does only allow a single process maintaining a database connection, all our database access is funneled through a special database process. To access the database from anoth...
Christoph
2

votes
1

answer
42

Views

How to subprocess a big list of files using all CPUs?

I need to convert 86,000 TEX files to XML using the LaTeXML library in the command line. I tried to write a Python script to automate this with the subprocess module, utilizing all 4 cores. def get_outpath(tex_path): path_parts = pathlib.Path(tex_path).parts arxiv_id = path_parts[2] outpath = 'xml/'...
briennakh
0

votes
0

answer
7

Views

Multiprocessing shared value issue with nested dict and slow function

While adjusting adjusting a value within a nested dictionary using multiprocessing, if there is a long process after copying the nested dictionary over the value isn't being updated and it looks like it's overwriting the other one. I've looked on various threads across stackoverflow, but couldn't fi...
J. Doe
-1

votes
1

answer
12

Views

How to pass a generator to create threading.Thread and get access on elements of generator in an array of parameters

In python2, I would like to launch 16 threading.Thread from import threading library. For each launch, I try to pass as parameters a generator to the function buildCrossMatrix_loop which is the target function : Here the generator : dimBlocks = 100 def generatorCrossMatrix(index): for igen in range(...
youpilat13
0

votes
0

answer
19

Views

How to run an object whose functions can be called as threads

I'm writing a script for a Raspberry Pi in Python whose purpose is to listen to a server/message broker for commands and execute said commands with certain hardware. Sometimes, those commands must last for a specified duration (i.e. I need something to turn on, stay on for t seconds, then turn off)...
Jaime B.
1

votes
1

answer
97

Views

How can I get values from Selenium WebElement more than faster?

This is my test code to get value from Selenium WebElement. import java.util.List; import org.apache.commons.lang3.ObjectUtils.Null; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebDriverException; import org.openqa.selenium.WebElement; import o...
Nori
0

votes
0

answer
19

Views

Why is there an EOF error when I try multiprocessing?

In my text game, I was using threads for a real-time battle system. It worked, but it asked you for a load of inputs at the end if you were too slow at answering. I tried to learn multiprocessing so I could terminate the threads and get rid of the extraneous inputs, but this made things worse. Now I...
0

votes
1

answer
21

Views

Passing second parameter to function in pool.map

I have a panda dataframe with many rows, I am using multiprocessing to process grouped tables from this dataframe concurrently. It works fine but I have a problem passing in a second parameter, I have tried to pass both arguments as a Tuple but it doesn't work. My code is as follows: I want to also...
Supes38
1

votes
2

answer
1.6k

Views

Deadlock with multiprocessing module

I have a function that without multiprocessing loops over an array with 3-tuples and does some calculation. This array can be really long (>1million entries) so I thought using several processes could help speed things up. I start with a list of points (random_points) with which I create a permutati...
Philipp
0

votes
0

answer
5

Views

Multiprocessing to build a single minibatch (Keras - TF backend)

In Keras API, there is the option to use Sequence to generate mini-batches in multi-process mode. However, the generation of a single minibatch remains a single process: i.e one needs to loop through a fixed set of examples (of size BATCH_SIZE) one at a time, perform data augmentation if needed, and...
JMarc
2

votes
1

answer
31

Views

Multiprocessing Pool.apply executing n-1 times

Im having an issues with multiprocessing.Pool.apply. My objective is to have 5 processes, each filling an array with 100 elements (100 for this test), and then merging then arrays into a single one with length 500. Problem is, it ends up with only 400 elements for any reason i cant understand. I hav...
chibi
1

votes
2

answer
1.1k

Views

What's the best way to design message key in Kafka?

I have a partitioned topic, which has X partitions. As of now, when producing messages, I create Kafka's ProducerRecord specifying only topic and value. I do not define a key. As far as I understand, my messages gonna be distributed evenly amongst partitions using default built-in partitioner. On t...
Ihor M.
1

votes
1

answer
308

Views

Why should one use tf.train.Server to execute multiple tf.Session() in parallel?

The official way to execute multiple tf.Session() in parallel is to use tf.train.Server as described in Distributed TensorFlow . On the other hand, the following works for Keras and can be modified to Tensorflow presumably without using tf.train.Server according to Keras + Tensorflow and Multiproces...
Math.StackExchange
1

votes
2

answer
1.2k

Views

How to run multiple webdriver python programs at the same time?

I have 4 Python scripts (3 web drivers and a main script). I would like to open these 3 web drivers at the same time when I run the mainscript.py. I used multiprocessing but you can use whatever you want. Now it opens bot_1.py and then bot_2.py and then bot_3.py. bot_1.py from selenium import webdri...
Andrei Ghervan
1

votes
1

answer
456

Views

Multiprocessing — Thread Pool Memory Leak?

I am observing memory usage that I cannot explain to myself. Below I provide a stripped down version of my actual code that still exhibits this behavior. The code is intended to accomplish the following: Read a text file in chunks of 1000 lines. Each line is a sentence. Split these 1000 sentences in...
lo tolmencre
1

votes
1

answer
715

Views

Python ValueError: Pool not running in Async Multiprocessing

I have a simple code: path = [filepath1, filepath2, filepath3] def umap_embedding(filepath): file = np.genfromtxt(filepath,delimiter=' ') if len(file) > 20000: file = file[np.random.choice(file.shape[0], 20000, replace=False), :] neighbors = len(file)//200 if neighbors >= 2: neighbors = neighbors el...
Thanh Nguyen
1

votes
1

answer
82

Views

Python, invoke a process pool without blocking the event loop

If I run the following code: import asyncio import time import concurrent.futures def cpu_bound(mul): for i in range(mul*10**8): i+=1 print('result = ', i) return i async def say_after(delay, what): print('sleeping async...') await asyncio.sleep(delay) print(what) # The run_in_pool function must no...
dimyG
0

votes
0

answer
10

Views

Making many figures with multiprocessing causes text to be scrambled

I am using matplotlib with multiprocessing to make many figures quickly. The issue is that the text in the title on some figures is messed up. Some titles are messed up, like this... And others look correct... Has anyone else ever had this problem, and is there a way to prevent it from happening? F...
blaylockbk
1

votes
1

answer
1.4k

Views

shared objects after os.fork()

I've encountered some strange application behaviour while interacting with database using many processes. I'm using Linux. I have my own implementation of QueryExecutor which uses the a single connection during its lifetime: class QueryExecutor(object): def __init__(self, db_conf): self._db_config =...
Grzes
1

votes
1

answer
227

Views

python multiprocessing peculiar memory management

I have a simple multiprocessing code: from multiprocessing import Pool import time def worker(data): time.sleep(20) if __name__ == "__main__": numprocs = 10 pool = Pool(numprocs) a = ['a' for i in xrange(1000000)] b = [a+[] for i in xrange(100)] data1 = [b+[] for i in range(numprocs)] data2 = [dat...
navidoo
1

votes
1

answer
630

Views

Python Queue suddenly empty

I'm trying to run a couple of calculations concurrently based on this example. I extended it in the way that instead of just a simple function, I'm running some external software via subprocess. It's run with Python 2.7.6 on Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-30-generic x86_64). There is a Queue i...
helm
1

votes
1

answer
213

Views

python and postgres always share single CPU core?

I'm tinkering with some big-data queries in the ipython shell using the Django ORM. This is on a Debian 6 VM in VMware Fusion on OS X, the VM is allowed access 4 or 8 cores (I've played with the settings) of the 4-core HT i7 on the host. When I watch the progress in top, when doing for example a 'f...
taiganaut
1

votes
1

answer
205

Views

In Multiprocessing, why is 1 process faster than multiple when accessing this Queue?

I have this code and have noticed when specifying more than 1 thread, performance suffers. The concept here is simple, setup a Queue in multiprocessing, access it from each thread, equally-- note I am using an easily divisible queue size here. import time from multiprocessing import Process, Queue...
scape
1

votes
4

answer
409

Views

How to take and handle object from multiple threads in Java? How to create Multiple instances of Future

I need to create bunch of JMX connections to different hosts. I'm trying to create these connections in parallel. In this performance is much better now. I've method to which I'm passing Collection of "host and port" entries. For each entry one connection will be created. Now I'm submitting this tas...
Akshay Hiremath
1

votes
1

answer
931

Views

Python Multiprocessing Queue file descriptor

I wanted to know if the queue had a file descriptor and how to find it?
WojonsTech
1

votes
1

answer
914

Views

function initialize and object initialization (multiprocessing)

I recently saw a answer/comment about how functions are objects in python. So, I wondering why when I take that example, and create a class around it while initializing a variable, it doesn't work the same way. (The class example receives a pickling error): PicklingError: Can't pickle : attribute l...
chase
1

votes
1

answer
990

Views

multiprocessing.Process - why doesn't the .start() method work in the IDLE console?

import multiprocessing def simple(): print 'simple!' p0 = multiprocessing.Process(target=simple) p0.start() This doesn't do anything apparently. AFAIK it should print "simple!". I'm writing all this code inside the interactive console. [EDIT] Could the other process not be connected to this termina...
vlad-ardelean
1

votes
1

answer
1.1k

Views

python map_async, where is the overhead coming from?

I am using map_async as intended - to map an iterable over multiple processing cores using: cores = mp.cpu_count() pool = mp.Pool() r = pool.map_async(func, offsets,callback=mycallback) r.wait() func returns a dict, so the callback 'merges' the dicts using: ddict = defaultdict(set) def mycallback(w)...
Jzl5325
1

votes
2

answer
113

Views

Does last_insert_id return the correct auto_increment id in a multiprocessing environment?

Here's some simplified code in my web application: sub insert { my $pid = fork(); if ($pid > 0) { return; } else { &insert_to_mysql(); my $last_id = &get_last_inserted(); # call mysql last_inserted_id exit(0); } } for my $i (1..10) { &insert(); } Since insert is called in a multiprocessing environme...
1

votes
1

answer
1.1k

Views

Train two models concurrently

All I need to do is, train two regression models (using scikit-learn) on the same data at the same time, using different cores. I've tried to figured out by myself using Process without success. gb1 = GradientBoostingRegressor(n_estimators=10) gb2 = GradientBoostingRegressor(n_estimators=100) def t...
1

votes
1

answer
1k

Views

How do I start a http server and then open a web browser?

I´m trying this: import multiprocessing from wsgiref.simple_server import make_server import webbrowser import time def application(environ, start_response): start_response("200 OK", [("Content-Type", "text/plain")]) return ["Hello!"] class Server(multiprocessing.Process): def run(self): print "...
Txema
1

votes
1

answer
268

Views

Global variable doesn't save correctly when set from a Process()

In this Flask app, the page calls check() every 2 seconds and prints the current app.config['count'] to stdout. There's also a button that will spawn a process and add one to the global count -- when checked from inside the same function, it works correctly, but when check() tries, it's always stuck...
user1814016
1

votes
1

answer
1.5k

Views

Python kqueue with multiple events

I'm trying to write a little program to send and receive UDP traffic and receive commands via an HTTP interface. The HTTP server lives in one multiprocessing.Process; the UDP server lives in another. The two processes communicate via a python multiprocessing.Pipe. I've attached the complete code bel...
nflacco
1

votes
1

answer
1.9k

Views

how do I generate multiprocessing jobs inside a list without making duplicates?

How do I make a multiprocessor system work, which makes new jobs inside a list? I keep getting: assert self._popen is None, 'cannot start a process twice' AttributeError: 'Worker' object has no attribute '_popen' which makes sense, because I'm basically making multiple instances of the same job......
AlexW.H.B.
1

votes
2

answer
1.2k

Views

Logging separate files for different Processes in Python

I've seen a few questions regarding putting logs from different processes together when using the multiprocessing module in Python. I would like to do the opposite, produce separate log files for different processes, and they should log everything that happens when calling other modules without bein...
x-A-x
1

votes
3

answer
192

Views

Python Multiprocessing, Need to Give Extra Arguement

In Python, what do you do if you are using a multiprocessing and you need to give the function an extra agruement? Example: if value == "Y": pool = multiprocessing.Pool(processes=8) pool.map(verify_headers, url_list)
dman

View additional questions