Yes - multiprocessing forks the python interpreter. As long as you use the synchronous API, no Python is invoked inside the client background pools - just the C++ handler. That means the GIL isn't touched, which means the deadlocks go away.
I'm very happy with my Python scripts that use multiprocessing + sync APIs. That's definitely the direction for @dabercro.
Yeah - CMS still uses the async API and we still have odds-n-ends issues with locks on the error recovery paths. It's incredibly tricky and I wouldn't advice folks to go down that path if they weren't already there. I do think it'd improve if the error callbacks were done on the background thread pool.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/xrootd/xrootd","title":"xrootd/xrootd","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/xrootd/xrootd"}},"updates":{"snippets":[{"icon":"PERSON","message":"@bbockelm in #462: Yes - `multiprocessing` forks the python interpreter. As long as you use the synchronous API, no Python is invoked inside the client background pools - just the C++ handler. That means the GIL isn't touched, which means the deadlocks go away.\r\n\r\nI'm very happy with my Python scripts that use `multiprocessing` + `sync` APIs. That's definitely the direction for @dabercro.\r\n\r\nYeah - CMS still uses the async API and we still have odds-n-ends issues with locks on the error recovery paths. It's incredibly tricky and I wouldn't advice folks to go down that path if they weren't already there. I do think it'd improve if the error callbacks were done on the background thread pool."}],"action":{"name":"View Issue","url":"https://github.com/xrootd/xrootd/issues/462#issuecomment-282374535"}}}
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
