How to use multiprocessing queue in Python?

| | |

I"m having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let"s call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.

My main problem is that I really don"t know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)

How to use multiprocessing queue in Python? iat: Questions

iat

InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately

3 answers

Tried to perform REST GET through python requests with the following code and I got error.

Code snip:

import requests
header = {"Authorization": "Bearer..."}
url = az_base_url + az_subscription_id + "/resourcegroups/Default-Networking/resources?" + az_api_version
r = requests.get(url, headers=header)

Error:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79: 
          InsecurePlatformWarning: A true SSLContext object is not available. 
          This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. 
          For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

My python version is 2.7.3. I tried to install urllib3 and requests[security] as some other thread suggests, I still got the same error.

Wonder if anyone can provide some tips?

334

Answer #1

The docs give a fair indicator of what"s required., however requests allow us to skip a few steps:

You only need to install the security package extras (thanks @admdrew for pointing it out)

$ pip install requests[security]

or, install them directly:

$ pip install pyopenssl ndg-httpsclient pyasn1

Requests will then automatically inject pyopenssl into urllib3


If you"re on ubuntu, you may run into trouble installing pyopenssl, you"ll need these dependencies:

$ apt-get install libffi-dev libssl-dev

iat

Dynamic instantiation from string name of a class in dynamically imported module?

3 answers

In python, I have to instantiate certain class, knowing its name in a string, but this class "lives" in a dynamically imported module. An example follows:

loader-class script:

import sys
class loader:
  def __init__(self, module_name, class_name): # both args are strings
    try:
      __import__(module_name)
      modul = sys.modules[module_name]
      instance = modul.class_name() # obviously this doesn"t works, here is my main problem!
    except ImportError:
       # manage import error

some-dynamically-loaded-module script:

class myName:
  # etc...

I use this arrangement to make any dynamically-loaded-module to be used by the loader-class following certain predefined behaviours in the dyn-loaded-modules...

222

Answer #1

You can use getattr

getattr(module, class_name)

to access the class. More complete code:

module = __import__(module_name)
class_ = getattr(module, class_name)
instance = class_()

As mentioned below, we may use importlib

import importlib
module = importlib.import_module(module_name)
class_ = getattr(module, class_name)
instance = class_()

iat

How to get all of the immediate subdirectories in Python

3 answers

I"m trying to write a simple Python script that will copy a index.tpl to index.html in all of the subdirectories (with a few exceptions).

I"m getting bogged down by trying to get the list of subdirectories.

184

Answer #1

import os
def get_immediate_subdirectories(a_dir):
    return [name for name in os.listdir(a_dir)
            if os.path.isdir(os.path.join(a_dir, name))]

How to use multiprocessing queue in Python? sep: Questions

sep

How to print number with commas as thousands separators?

5 answers

I am trying to print an integer in Python 2.6.1 with commas as thousands separators. For example, I want to show the number 1234567 as 1,234,567. How would I go about doing this? I have seen many examples on Google, but I am looking for the simplest practical way.

It does not need to be locale-specific to decide between periods and commas. I would prefer something as simple as reasonably possible.

929

Answer #1

Locale unaware

"{:,}".format(value)  # For Python ‚â•2.7
f"{value:,}"  # For Python ‚â•3.6

Locale aware

import locale
locale.setlocale(locale.LC_ALL, "")  # Use "" for auto, or force e.g. to "en_US.UTF-8"

"{:n}".format(value)  # For Python ‚â•2.7
f"{value:n}"  # For Python ‚â•3.6

Reference

Per Format Specification Mini-Language,

The "," option signals the use of a comma for a thousands separator. For a locale aware separator, use the "n" integer presentation type instead.

929

Answer #2

I got this to work:

>>> import locale
>>> locale.setlocale(locale.LC_ALL, "en_US")
"en_US"
>>> locale.format("%d", 1255000, grouping=True)
"1,255,000"

Sure, you don"t need internationalization support, but it"s clear, concise, and uses a built-in library.

P.S. That "%d" is the usual %-style formatter. You can have only one formatter, but it can be whatever you need in terms of field width and precision settings.

P.P.S. If you can"t get locale to work, I"d suggest a modified version of Mark"s answer:

def intWithCommas(x):
    if type(x) not in [type(0), type(0L)]:
        raise TypeError("Parameter must be an integer.")
    if x < 0:
        return "-" + intWithCommas(-x)
    result = ""
    while x >= 1000:
        x, r = divmod(x, 1000)
        result = ",%03d%s" % (r, result)
    return "%d%s" % (x, result)

Recursion is useful for the negative case, but one recursion per comma seems a bit excessive to me.

sep

How would you make a comma-separated string from a list of strings?

5 answers

mweerden By mweerden

What would be your preferred way to concatenate strings from a sequence such that between every two consecutive pairs a comma is added. That is, how do you map, for instance, ["a", "b", "c"] to "a,b,c"? (The cases ["s"] and [] should be mapped to "s" and "", respectively.)

I usually end up using something like "".join(map(lambda x: x+",",l))[:-1], but also feeling somewhat unsatisfied.

612

Answer #1

my_list = ["a", "b", "c", "d"]
my_string = ",".join(my_list)
"a,b,c,d"

This won"t work if the list contains integers


And if the list contains non-string types (such as integers, floats, bools, None) then do:

my_string = ",".join(map(str, my_list)) 

Shop

Best laptop for Fortnite

$

Best laptop for Excel

$

Best laptop for Solidworks

$

Best laptop for Roblox

$

Best computer for crypto mining

$

Best laptop for Sims 4

$

Best laptop for Zoom

$499

Best laptop for Minecraft

$590

Latest questions

NUMPYNUMPY

psycopg2: insert multiple rows with one query

12 answers

NUMPYNUMPY

How to convert Nonetype to int or string?

12 answers

NUMPYNUMPY

How to specify multiple return types using type-hints

12 answers

NUMPYNUMPY

Javascript Error: IPython is not defined in JupyterLab

12 answers

News

Wiki

Python OpenCV | cv2.putText () method

numpy.arctan2 () in Python

Python | os.path.realpath () method

Python OpenCV | cv2.circle () method

Python OpenCV cv2.cvtColor () method

Python - Move item to the end of the list

time.perf_counter () function in Python

Check if one list is a subset of another in Python

Python os.path.join () method