Step-by-step debugging with IPython

breakpoint | StackOverflow

From what I have read, there are two ways to debug code in Python:

  • With a traditional debugger such as pdb or ipdb. This supports commands such as c for continue, n for step-over, s for step-into etc.), but you don"t have direct access to an IPython shell which can be extremely useful for object inspection.

  • Using IPython by embedding an IPython shell in your code. You can do from IPython import embed, and then use embed() in your code. When your program/script hits an embed() statement, you are dropped into an IPython shell. This allows the full inspection of objects and testing of Python code using all the IPython goodies. However, when using embed() you can"t step-by-step through the code anymore with handy keyboard shortcuts.

Is there any way to combine the best of both worlds? I.e.

  1. Be able to step-by-step through your code with handy pdb/ipdb keyboard shortcuts.
  2. At any such step (e.g. on a given statement), have access to a full-fledged IPython shell.

IPython debugging as in MATLAB:

An example of this type of "enhanced debugging" can be found in MATLAB, where the user always has full access to the MATLAB engine/shell, and she can still step-by-step through her code, define conditional breakpoints, etc. From what I have discussed with other users, this is the debugging feature that people miss the most when moving from MATLAB to IPython.

IPython debugging in Emacs and other editors:

I don"t want to make the question too specific, but I work mostly in Emacs, so I wonder if there is any way to bring this functionality into it. Ideally, Emacs (or the editor) would allow the programmer to set breakpoints anywhere on the code and communicate with the interpreter or debugger to have it stop in the location of your choice, and bring to a full IPython interpreter on that location.

Answer rating: 75

You can use IPython"s %pdb magic. Just call %pdb in IPython and when an error occurs, you"re automatically dropped to ipdb. While you don"t have the stepping immediately, you"re in ipdb afterwards.

This makes debugging individual functions easy, as you can just load a file with %load and then run a function. You could force an error with an assert at the right position.

%pdb is a line magic. Call it as %pdb on, %pdb 1, %pdb off or %pdb 0. If called without argument it works as a toggle.





Step-by-step debugging with IPython: StackOverflow Questions

Simpler way to put PDB breakpoints in Python code?

Just a convenience question. I"ve been a bit spoiled with debuggers in IDEs like Visual Studio and XCode. I find it a bit clumsy to have to type import pdb; pdb.set_trace() to set a breakpoint (I"d rather not import pdb at the top of the file as I might forget and leave it in).

Is there a simpler way of setting a breakpoint in Python code, as straightforward and unobtrusive as what you see in an IDE?

Answer #1

To begin, note that quantiles is just the most general term for things like percentiles, quartiles, and medians. You specified five bins in your example, so you are asking qcut for quintiles.

So, when you ask for quintiles with qcut, the bins will be chosen so that you have the same number of records in each bin. You have 30 records, so should have 6 in each bin (your output should look like this, although the breakpoints will differ due to the random draw):

pd.qcut(factors, 5).value_counts()

[-2.578, -0.829]    6
(-0.829, -0.36]     6
(-0.36, 0.366]      6
(0.366, 0.868]      6
(0.868, 2.617]      6

Conversely, for cut you will see something more uneven:

pd.cut(factors, 5).value_counts()

(-2.583, -1.539]    5
(-1.539, -0.5]      5
(-0.5, 0.539]       9
(0.539, 1.578]      9
(1.578, 2.617]      2

That"s because cut will choose the bins to be evenly spaced according to the values themselves and not the frequency of those values. Hence, because you drew from a random normal, you"ll see higher frequencies in the inner bins and fewer in the outer. This is essentially going to be a tabular form of a histogram (which you would expect to be fairly bell shaped with 30 records).

Answer #2

You can use ipdb inside jupyter with:

from IPython.core.debugger import Tracer; Tracer()()

Edit: the functions above are deprecated since IPython 5.1. This is the new approach:

from IPython.core.debugger import set_trace

Add set_trace() where you need a breakpoint. Type help for ipdb commands when the input field appears.

Answer #3

By using Python Interactive Debugger "pdb"

First step is to make the Python interpreter to enter into the debugging mode.

A. From the Command Line

Most straight forward way, running from command line, of python interpreter

$ python -m pdb scriptName.py
> .../pdb_script.py(7)<module>()
-> """
(Pdb)

B. Within the Interpreter

While developing early versions of modules and to experiment it more iteratively.

$ python
Python 2.7 (r27:82508, Jul  3 2010, 21:12:11)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb_script
>>> import pdb
>>> pdb.run("pdb_script.MyObj(5).go()")
> <string>(1)<module>()
(Pdb)

C. From Within Your Program

For a big project and long-running module, can start the debugging from inside the program using import pdb and set_trace() like this :

#!/usr/bin/env python
# encoding: utf-8
#

import pdb

class MyObj(object):
    count = 5
    def __init__(self):
        self.count= 9

    def go(self):
        for i in range(self.count):
            pdb.set_trace()
            print i
        return

if __name__ == "__main__":
    MyObj(5).go()

Step-by-Step debugging to go into more internal

  1. Execute the next statement… with “n” (next)

  2. Repeating the last debugging command… with ENTER

  3. Quitting it all… with “q” (quit)

  4. Printing the value of variables… with “p” (print)

    a) p a

  5. Turning off the (Pdb) prompt… with “c” (continue)

  6. Seeing where you are… with “l” (list)

  7. Stepping into subroutines… with “s” (step into)

  8. Continuing… but just to the end of the current subroutine… with “r” (return)

  9. Assign a new value

    a) !b = "B"

  10. Set a breakpoint

    a) break linenumber

    b) break functionname

    c) break filename:linenumber

  11. Temporary breakpoint

    a) tbreak linenumber

  12. Conditional breakpoint

    a) break linenumber, condition

Note:**All these commands should be execute from **pdb

For in-depth knowledge, refer:-

https://pymotw.com/2/pdb/

https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

Answer #4

1 - To answer your question directly:

The single most important difference between the two is that you should start using JupyterLab straight away, and that you should not worry about Jupyter Notebook at all. Because:

JupyterLab will eventually replace the classic Jupyter Notebook. Throughout this transition, the same notebook document format will be supported by both the classic Notebook and JupyterLab

As of version 3.0, JupyterLab also comes with a visual debugger that lets you interactively set breakpoints, step into functions, and inspect variables.

2 - To contradict the numerous claims in the comments that plotly does not run well with JLab:

JupyterLab is an absolutely fantastic tool both to build plotly figures, and fire up complete Dash Apps both inline, as a tab, and externally in a browser.

3 - And you would probably also like to know this:

Other posts have suggested that Jupyter Notebook (JN) could potentially be easier to use than JupyterLab (JL) for beginners. But I would have to disagree.

A great advantage with JL, and arguably one of the most important differences between JL and JN, is that you can more easily run a single line and even highlighted text. I prefer using a keyboard shortcut for this, and assigning shortcuts is pretty straight-forward.

And the fact that you can execute code in a Python console makes JL much more fun to work with. Other answers have already mentioned this, but JL can in some ways be considered a tool to run Notebooks and more. So the way I use JupyterLab is by having it set up with an .ipynb file, a file browser and a python console like this:

enter image description here

And now you have these tools at your disposal:

  1. View Files, running kernels, Commands, Notebook Tools, Open Tabs or Extension manager
  2. Run cells using, among other options, Ctrl+Enter
  3. Run single expression, line or highlighted text using menu options or keyboard shortcuts
  4. Run code directly in a console using Shift+Enter
  5. Inspect variables, dataframes or plots quickly and easily in a console without cluttering your notebook output.

Answer #5

There are a bunch of ways to do it, but the most straightforward is to simply use the Python debugger. Just add following line in to a Django view function:

import pdb; pdb.set_trace()

or

breakpoint()  #from Python3.7

If you try to load that page in your browser, the browser will hang and you get a prompt to carry on debugging on actual executing code.

However there are other options (I am not recommending them):

* return HttpResponse({variable to inspect})

* print {variable to inspect}

* raise Exception({variable to inspect})

But the Python Debugger (pdb) is highly recommended for all types of Python code. If you are already into pdb, you"d also want to have a look at IPDB that uses ipython for debugging.

Some more useful extension to pdb are

pdb++, suggested by Antash.

pudb, suggested by PatDuJour.

Using the Python debugger in Django, suggested by Seafangs.

Answer #6

Yes! There"s a Python debugger called pdb just for doing that!

You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py.

There are a few commands you can then issue, which are documented on the pdb page.

Some useful ones to remember are:

  • b: set a breakpoint
  • c: continue debugging until you hit a breakpoint
  • s: step through the code
  • n: to go to next line of code
  • l: list source code for the current file (default: 11 lines including the line being executed)
  • u: navigate up a stack frame
  • d: navigate down a stack frame
  • p: to print the value of an expression in the current context

If you don"t want to use a command line debugger, some IDEs like Pydev, Wing IDE or PyCharm have a GUI debugger. Wing and PyCharm are commercial products, but Wing has a free "Personal" edition, and PyCharm has a free community edition.

Answer #7

continue should "Continue execution, only stop when a breakpoint is encountered", so you"ve got a breakpoint set somewhere. To remove the breakpoint (if you inserted it manually):

(Pdb) break
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at /path/to/test.py:5
(Pdb) clear 1
Deleted breakpoint 1
(Pdb) continue

Or, if you"re using pdb.set_trace(), you can try this (although if you"re using pdb in more fancy ways, this may break things...)

(Pdb) pdb.set_trace = lambda: None  # This replaces the set_trace() function!
(Pdb) continue
# No more breaks!

Answer #8

The scrapy command is a python script which means you can start it from inside PyCharm.

When you examine the scrapy binary (which scrapy) you will notice that this is actually a python script:

#!/usr/bin/python

from scrapy.cmdline import execute
execute()

This means that a command like scrapy crawl IcecatCrawler can also be executed like this: python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler

Try to find the scrapy.cmdline package. In my case the location was here: /Library/Python/2.7/site-packages/scrapy/cmdline.py

Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case crawl IcecatCrawler.

Like this: PyCharm Run/Debug Configuration

Put your breakpoints anywhere in your crawling code and it should work‚Ñ¢.

Answer #9

Running the app in development mode will show an interactive traceback and console in the browser when there is an error. To run in development mode, set the FLASK_ENV=development environment variable then use the flask run command (remember to point FLASK_APP to your app as well).

For Linux, Mac, Linux Subsystem for Windows, Git Bash on Windows, etc.:

export FLASK_APP=myapp
export FLASK_ENV=development
flask run

For Windows CMD, use set instead of export:

set FLASK_ENV=development

For PowerShell, use $env:

$env:FLASK_ENV = "development"

Prior to Flask 1.0, this was controlled by the FLASK_DEBUG=1 environment variable instead.

If you"re using the app.run() method instead of the flask run command, pass debug=True to enable debug mode.

Tracebacks are also printed to the terminal running the server, regardless of development mode.

If you"re using PyCharm, VS Code, etc., you can take advantage of its debugger to step through the code with breakpoints. The run configuration can point to a script calling app.run(debug=True, use_reloader=False), or point it at the venv/bin/flask script and use it as you would from the command line. You can leave the reloader disabled, but a reload will kill the debugging context and you will have to catch a breakpoint again.

You can also use pdb, pudb, or another terminal debugger by calling set_trace in the view where you want to start debugging.


Be sure not to use too-broad except blocks. Surrounding all your code with a catch-all try... except... will silence the error you want to debug. It"s unnecessary in general, since Flask will already handle exceptions by showing the debugger or a 500 error and printing the traceback to the console.

Tutorials