Parallel Threads in Python &Daemon thread

daemon thread | Python Methods and Functions

In today's post, we'll take a look at using streams in Python. Before we start with the main topic, let's first take a look at what Daemon Thread is.

 Daemon Thread




What is Daemon Thread?

Daemon Thread in Python &it is a type of thread that can run independently in the background. These kinds of threads run independently of the main thread. So they are called non-blocking streams.






When do you need Daemon streams in Python? < / h2>

Suppose you need to have a long running task that tries to read log files. This task should alert the user when an error message is encountered in the logs.

We can assign a Daemon thread for this task, which can monitor our log files while our main program does its normal work.

Most useful things about Daemon streams &it is that they automatically stop execution after the main program finishes.

If you need a short task, the Daemon thread will stop executing after it returns. However, due to this nature, threads are widely used for long-running background tasks.




Practical implementation

These Python examples will use the streaming module, which is part of the standard library. 

 import threading 

To illustrate the power of threads, let's first create two threads, A and B.



We will force thread A to perform a short computation, while thread B will try to track the share.

If this resource is set to True , we will force thread B to warn the user about the status.

 impo rt threading import time # Set the resource to False initially shared_resource = False # A lock for the shared resource lock = threading.Lock () def perform_computation (): # Thread A will call this function and manipulate the resource print (f'Thread {threading.currentThread () .name} - performing some computation .... ') shared_resource = True print (f'Thread {threading.currentThread (). name} - set shared_resource to True!') print (f'Thread {threading.currentThread (). name} - Finished! ') time.sleep (1) def monitor_re source (): # Thread B will monitor the shared resource while shared_resource == False: time.sleep (1) print (f'Thread {threading.currentThread (). name} - Detected shared_resource = False') time.sleep (1) print (f'Thread {threading.currentThread (). name} - Finished! ') if __name__ ==' __main__': a = threading.Thread (target = perform_computation, name = 'A') b = threading.Thread (target = monitor_resource, name =' B') # Now start both threads a.start () b.start () 

Here thread A will set the shared_resource to True and stream B will wait until this resource becomes True.

Exit

 Thread A - performing some computation .... Thread A - set shared_resource to True! Thread A - Finished! Thread B - Detected shared_resource = False Thread B - Finished! 

Note that both threads are regular threads.

Now let's make thread B a daemon thread. Let's see what happens now. To do this, we can set it as a parameter in the threading.Thread (daemon = True) method.

 import threading import time shared_resource = False # Set the resource to False initially lock = threading.Lock () # A lock for the shared resource def perform_computation (): # Thread A will call this function and manipulate the resource print (f'Thread {threading.currentThread () .name} - performing some computation .... ') shared_resource = True print (f'Thread {threading.currentThread (). name} - set shared_resource to True!') print (f'Thread {threading.currentThread ( ) .name} - Finished! ') time.sleep (1) def monitor_resource (): # Thread B will monitor the shared resource while shared_resource == False: time.sle ep (1) print (f'Daemon Thread {threading. currentThread (). name} - Detected shared_resource = False') time.sleep (1) print (f'Daemon Thread {threading.currentThread (). name} - Finished! ') if __name__ ==' __main__': a = threading .T hread (target = perform_computation, name = 'A') b = threading.Thread (target = monitor_resource, name =' B', daemon = True) # Make thread B as a daemon thread # Now start both threads a.start () b.start () 

Exit

 Thread A - performing some computation .... Thread A - set shared_resource to True! Thread A - Finished! Daemon Thread B - Detected shared_resource = False 

Please note that Daemon Thread does not terminate. This is because it will be automatically killed by the main thread. The non-blocking nature of threads makes them very useful for many Python applications.







Parallel Threads in Python &Daemon thread: StackOverflow Questions

Daemon Threads Explanation

Question by cgoldberg

In the Python documentation it says:

A thread can be flagged as a "daemon thread". The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread.

Does anyone have a clearer explanation of what that means or a practical example showing where you would set threads as daemonic?

Clarify it for me: so the only situation you wouldn"t set threads as daemonic, is when you want them to continue running after the main thread exits?

Answer #1

"Guaranteed" is a much stronger word than any implementation of finally deserves. What is guaranteed is that if execution flows out of the whole try-finally construct, it will pass through the finally to do so. What is not guaranteed is that execution will flow out of the try-finally.

  • A finally in a generator or async coroutine might never run, if the object never executes to conclusion. There are a lot of ways that could happen; here"s one:

    def gen(text):
        try:
            for line in text:
                try:
                    yield int(line)
                except:
                    # Ignore blank lines - but catch too much!
                    pass
        finally:
            print("Doing important cleanup")
    
    text = ["1", "", "2", "", "3"]
    
    if any(n > 1 for n in gen(text)):
        print("Found a number")
    
    print("Oops, no cleanup.")
    

    Note that this example is a bit tricky: when the generator is garbage collected, Python attempts to run the finally block by throwing in a GeneratorExit exception, but here we catch that exception and then yield again, at which point Python prints a warning ("generator ignored GeneratorExit") and gives up. See PEP 342 (Coroutines via Enhanced Generators) for details.

    Other ways a generator or coroutine might not execute to conclusion include if the object is just never GC"ed (yes, that"s possible, even in CPython), or if an async with awaits in __aexit__, or if the object awaits or yields in a finally block. This list is not intended to be exhaustive.

  • A finally in a daemon thread might never execute if all non-daemon threads exit first.

  • os._exit will halt the process immediately without executing finally blocks.

  • os.fork may cause finally blocks to execute twice. As well as just the normal problems you"d expect from things happening twice, this could cause concurrent access conflicts (crashes, stalls, ...) if access to shared resources is not correctly synchronized.

    Since multiprocessing uses fork-without-exec to create worker processes when using the fork start method (the default on Unix), and then calls os._exit in the worker once the worker"s job is done, finally and multiprocessing interaction can be problematic (example).

  • A C-level segmentation fault will prevent finally blocks from running.
  • kill -SIGKILL will prevent finally blocks from running. SIGTERM and SIGHUP will also prevent finally blocks from running unless you install a handler to control the shutdown yourself; by default, Python does not handle SIGTERM or SIGHUP.
  • An exception in finally can prevent cleanup from completing. One particularly noteworthy case is if the user hits control-C just as we"re starting to execute the finally block. Python will raise a KeyboardInterrupt and skip every line of the finally block"s contents. (KeyboardInterrupt-safe code is very hard to write).
  • If the computer loses power, or if it hibernates and doesn"t wake up, finally blocks won"t run.

The finally block is not a transaction system; it doesn"t provide atomicity guarantees or anything of the sort. Some of these examples might seem obvious, but it"s easy to forget such things can happen and rely on finally for too much.

Answer #2

Here"s a simple example: you need to try a few alternative URLs and return the contents of the first one to respond.

import Queue
import threading
import urllib2

# Called by each thread
def get_url(q, url):
    q.put(urllib2.urlopen(url).read())

theurls = ["http://google.com", "http://yahoo.com"]

q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target=get_url, args = (q,u))
    t.daemon = True
    t.start()

s = q.get()
print s

This is a case where threading is used as a simple optimization: each subthread is waiting for a URL to resolve and respond, to put its contents on the queue; each thread is a daemon (won"t keep the process up if the main thread ends -- that"s more common than not); the main thread starts all subthreads, does a get on the queue to wait until one of them has done a put, then emits the results and terminates (which takes down any subthreads that might still be running, since they"re daemon threads).

Proper use of threads in Python is invariably connected to I/O operations (since CPython doesn"t use multiple cores to run CPU-bound tasks anyway, the only reason for threading is not blocking the process while there"s a wait for some I/O). Queues are almost invariably the best way to farm out work to threads and/or collect the work"s results, by the way, and they"re intrinsically threadsafe, so they save you from worrying about locks, conditions, events, semaphores, and other inter-thread coordination/communication concepts.

Answer #3

Some threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it"s okay to kill them off once the other, non-daemon, threads have exited.

Without daemon threads, you"d have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.

Answer #4

A somewhat clumsy ascii-art to demonstrate the mechanism: The join() is presumably called by the main-thread. It could also be called by another thread, but would needlessly complicate the diagram.

join-calling should be placed in the track of the main-thread, but to express thread-relation and keep it as simple as possible, I choose to place it in the child-thread instead.

without join:
+---+---+------------------                     main-thread
    |   |
    |   +...........                            child-thread(short)
    +..................................         child-thread(long)

with join
+---+---+------------------***********+###      main-thread
    |   |                             |
    |   +...........join()            |         child-thread(short)
    +......................join()......         child-thread(long)

with join and daemon thread
+-+--+---+------------------***********+###     parent-thread
  |  |   |                             |
  |  |   +...........join()            |        child-thread(short)
  |  +......................join()......        child-thread(long)
  +,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,     child-thread(long + daemonized)

"-" main-thread/parent-thread/main-program execution
"." child-thread execution
"#" optional parent-thread execution after join()-blocked parent-thread could 
    continue
"*" main-thread "sleeping" in join-method, waiting for child-thread to finish
"," daemonized thread - "ignores" lifetime of other threads;
    terminates when main-programs exits; is normally meant for 
    join-independent tasks

So the reason you don"t see any changes is because your main-thread does nothing after your join. You could say join is (only) relevant for the execution-flow of the main-thread.

If, for example, you want to concurrently download a bunch of pages to concatenate them into a single large page, you may start concurrent downloads using threads, but need to wait until the last page/thread is finished before you start assembling a single page out of many. That"s when you use join().

Answer #5

Ctrl+C terminates the main thread, but because your threads aren"t in daemon mode, they keep running, and that keeps the process alive. We can make them daemons:

f = FirstThread()
f.daemon = True
f.start()
s = SecondThread()
s.daemon = True
s.start()

But then there"s another problem - once the main thread has started your threads, there"s nothing else for it to do. So it exits, and the threads are destroyed instantly. So let"s keep the main thread alive:

import time
while True:
    time.sleep(1)

Now it will keep print "first" and "second" until you hit Ctrl+C.

Edit: as commenters have pointed out, the daemon threads may not get a chance to clean up things like temporary files. If you need that, then catch the KeyboardInterrupt on the main thread and have it co-ordinate cleanup and shutdown. But in many cases, letting daemon threads die suddenly is probably good enough.

Get Solution for free from DataCamp guru