OpenCV-Python is a Python link library designed to solve computer vision problems. The cv2.circle () method is used to draw a circle on any image.
Syntax: cv2.circle(image, center_coordinates, radius, color, thickness)
Parameters:
image: It is the image on which circle is to be drawn.
center_coordinates: It is the center coordinates of circle. The coordinates are represented as tuples of two values i.e. (X coordinate value, Y coordinate value).
radius: It is the radius of circle.
color: It is the color of border line of circle to be drawn. For BGR, we pass a tuple. eg: (255, 0, 0) for blue color.
thickness: It is the thickness of the circle border line in px. Thickness of -1 px will fill the circle shape by the specified color.
Return Value: It returns an image.
Im usign python and opencv to get a image from the webcam, and I want to know how to draw a circle over my image, just a simple green circle with transparent fill
my code:
import cv2
import numpy
import sys
if __name__ == '__main__':
#get current frame from webcam
cam = cv2.VideoCapture(0)
img = cam.read()
#how draw a circle????
cv2.imshow('WebCam', img)
cv2.waitKey()
Thanks in advance.
cv2.circle(img, center, radius, color, thickness=1, lineType=8, shift=0) → None
Draws a circle.
Parameters:
img (CvArr) – Image where the circle is drawn
center (CvPoint) – Center of the circle
radius (int) – Radius of the circle
color (CvScalar) – Circle color
thickness (int) – Thickness of the circle outline if positive, otherwise this indicates that a filled circle is to be drawn
lineType (int) – Type of the circle boundary, see Line description
shift (int) – Number of fractional bits in the center coordinates and radius value
Use "thickness" parameter for only the border.
def blob(x):
"""Given an Nx3 matrix of blob positions and size,
create N img_size x img_size images, each with a blob drawn on
them given by the value in each row of x
One row of x = [x,y,radius]."""
y = np.zeros((x.shape[0], img_size, img_size))
for i, particle in enumerate(x):
rr, cc = skimage.draw.circle(
particle[0], particle[1], max(particle[2], 1), shape=(img_size, img_size)
)
y[i, rr, cc] = 1
return y
#%%
# names (this is just for reference for the moment!)
def update(self, radarData):
self.img = np.zeros((self.height, self.width, self.channels), np.uint8)
cv2.line(self.img, (10, 0), (self.width/2 - 5, self.height), (100, 255, 255))
cv2.line(self.img, (self.width - 10, 0), (self.width/2 + 5, self.height), (100, 255, 255))
for track_number in range(1, 65):
if str(track_number)+'_track_range' in radarData:
track_range = radarData[str(track_number)+'_track_range']
track_angle = (float(radarData[str(track_number)+'_track_angle'])+90.0)*math.pi/180
x_pos = math.cos(track_angle)*track_range*4
y_pos = math.sin(track_angle)*track_range*4
cv2.circle(self.img, (self.width/2 + int(x_pos), self.height - int(y_pos) - 10), 5, (255, 255, 255))
#cv2.putText(self.img, str(track_number),
# (self.width/2 + int(x_pos)-2, self.height - int(y_pos) - 10), self.font, 1, (255,255,255), 2)
cv2.imshow("Radar", self.img)
cv2.waitKey(2)
def ProcessFrame(self, frame):
# segment arm region
segment = self.SegmentArm(frame)
# make a copy of the segmented image to draw on
draw = cv2.cvtColor(segment, cv2.COLOR_GRAY2RGB)
# draw some helpers for correctly placing hand
cv2.circle(draw,(self.imgWidth/2,self.imgHeight/2),3,[255,102,0],2)
cv2.rectangle(draw, (self.imgWidth/3,self.imgHeight/3), (self.imgWidth*2/3, self.imgHeight*2/3), [255,102,0],2)
# find the hull of the segmented area, and based on that find the
# convexity defects
[contours,defects] = self.FindHullDefects(segment)
# detect the number of fingers depending on the contours and convexity defects
# draw defects that belong to fingers green, others red
[nofingers,draw] = self.DetectNumberFingers(contours, defects, draw)
# print number of fingers on image
cv2.putText(draw, str(nofingers), (30,30), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255))
return draw
def mark_hand_center(frame_in,cont):
max_d=0
pt=(0,0)
x,y,w,h = cv2.boundingRect(cont)
for ind_y in xrange(int(y+0.3*h),int(y+0.8*h)): #around 0.25 to 0.6 region of height (Faster calculation with ok results)
for ind_x in xrange(int(x+0.3*w),int(x+0.6*w)): #around 0.3 to 0.6 region of width (Faster calculation with ok results)
dist= cv2.pointPolygonTest(cont,(ind_x,ind_y),True)
if(dist>max_d):
max_d=dist
pt=(ind_x,ind_y)
if(max_d>radius_thresh*frame_in.shape[1]):
thresh_score=True
cv2.circle(frame_in,pt,int(max_d),(255,0,0),2)
else:
thresh_score=False
return frame_in,pt,max_d,thresh_score
# 6. Find and display gesture
def update(self, radarData):
self.img = np.zeros((self.height, self.width, self.channels), np.uint8)
cv2.line(self.img, (10, 0), (self.width/2 - 5, self.height), (100, 255, 255))
cv2.line(self.img, (self.width - 10, 0), (self.width/2 + 5, self.height), (100, 255, 255))
for track_number in range(1, 65):
if str(track_number)+'_track_range' in radarData:
track_range = radarData[str(track_number)+'_track_range']
track_angle = (float(radarData[str(track_number)+'_track_angle'])+90.0)*math.pi/180
x_pos = math.cos(track_angle)*track_range*4
y_pos = math.sin(track_angle)*track_range*4
cv2.circle(self.img, (self.width/2 + int(x_pos), self.height - int(y_pos) - 10), 5, (255, 255, 255))
#cv2.putText(self.img, str(track_number),
# (self.width/2 + int(x_pos)-2, self.height - int(y_pos) - 10), self.font, 1, (255,255,255), 2)
cv2.imshow("Radar", self.img)
cv2.waitKey(2)
OpenCV-Python — is a Python bindings library for solving computer vision problems. cv2.circle () is used to draw a circle on any image.
Syntax: cv2.circle (image, center_coordinates, radius, color , thickness)
Parameters:
image: It is the image on which circle is to be drawn.
center_coordinates: It is the center coordinates of circle. The coordinates are represented as tuples of two values ie ( X coordinate value, Y coordinate value).
radius: It is the radius of circle.
color: It is the color of border line of circle to be drawn. For BGR , we pass a tuple. eg: (255, 0, 0) for blue color.
thickness: It is the thickness of the circle border line in px . Thickness of -1 px will fill the rectangle shape by the specified color.Return Value: It returns an image.
The image is used for all examples below:
Example # 1:
# Python program to explain the cv2.circle () method
# cv2 import
importcv2
# path
path=r'C: UsersRajnishDesktoppythonengineeringgeeks.png'
# Read image in default mode
image=cv2.imread (path)
# Name of the window in which the image is displayed
window_name='Image'
# Center coordinates
center_coordinates=(120,50)
# Circle radius
radius=20
# Blue color in BGR
color=(255,0,0)
# Line width 2 px
thickness=2
# Using the cv2.circle () method
# Draw a circle with a 2 px blue border
image=cv2.circle (image, center_coordinates, radius, color, thickness)
# Display image
cv2.imshow (window_name, image)
Output:
Example # 2:
Using -1 px thickness to fill the rectangle with red.
# Python program to explain the cv2.circle () method
# cv2 import
importcv2
# path
path=r'C: UsersRajnishDesktoppythonengineeringgeeks.png'
# Read image in default mode
image=cv2.imread (path)
# Name of the window in which the image is displayed
window_name='Image'
# Center coordinates
center_coordinates=(120,100)
# Circle radius
radius=30
# Red in BGR
color=(0,0,255)
# Line thickness -1 px
thickness=-1
# Using the cv2.circle () method
# Draw a -1 px red circle
image=cv2.circle (image, center_coordinates, radius, color, thickness)< code class = "undefined spaces">
# Displaying an image
cv2.imshow (window_name, image )
Output:
In Python, with Matplotlib, how can a scatter plot with empty circles be plotted? The goal is to draw empty circles around some of the colored disks already plotted by scatter(), so as to highlight them, ideally without having to redraw the colored circles.
I tried facecolors=None, to no avail.
surprisingly I didn"t find a straight-forward description on how to draw a circle with matplotlib.pyplot (please no pylab) taking as input center (x,y) and radius r. I tried some variants of this:
import matplotlib.pyplot as plt
circle=plt.Circle((0,0),2)
# here must be something like circle.plot() or not?
plt.show()
... but still didn"t get it working.
You said you couldn’t get the golden spiral method to work and that’s a shame because it’s really, really good. I would like to give you a complete understanding of it so that maybe you can understand how to keep this away from being “bunched up.”
So here’s a fast, non-random way to create a lattice that is approximately correct; as discussed above, no lattice will be perfect, but this may be good enough. It is compared to other methods e.g. at BendWavy.org but it just has a nice and pretty look as well as a guarantee about even spacing in the limit.
To understand this algorithm, I first invite you to look at the 2D sunflower spiral algorithm. This is based on the fact that the most irrational number is the golden ratio (1 + sqrt(5))/2 and if one emits points by the approach “stand at the center, turn a golden ratio of whole turns, then emit another point in that direction,” one naturally constructs a spiral which, as you get to higher and higher numbers of points, nevertheless refuses to have well-defined ‘bars’ that the points line up on.(Note 1.)
The algorithm for even spacing on a disk is,
from numpy import pi, cos, sin, sqrt, arange
import matplotlib.pyplot as pp
num_pts = 100
indices = arange(0, num_pts, dtype=float) + 0.5
r = sqrt(indices/num_pts)
theta = pi * (1 + 5**0.5) * indices
pp.scatter(r*cos(theta), r*sin(theta))
pp.show()
and it produces results that look like (n=100 and n=1000):
The key strange thing is the formula r = sqrt(indices / num_pts); how did I come to that one? (Note 2.)
Well, I am using the square root here because I want these to have even-area spacing around the disk. That is the same as saying that in the limit of large N I want a little region R ∈ (r, r + dr), Θ ∈ (θ, θ + dθ) to contain a number of points proportional to its area, which is r dr dθ. Now if we pretend that we are talking about a random variable here, this has a straightforward interpretation as saying that the joint probability density for (R, Θ) is just c r for some constant c. Normalization on the unit disk would then force c = 1/π.
Now let me introduce a trick. It comes from probability theory where it’s known as sampling the inverse CDF: suppose you wanted to generate a random variable with a probability density f(z) and you have a random variable U ~ Uniform(0, 1), just like comes out of random() in most programming languages. How do you do this?
Now the golden-ratio spiral trick spaces the points out in a nicely even pattern for θ so let’s integrate that out; for the unit disk we are left with F(r) = r2. So the inverse function is F-1(u) = u1/2, and therefore we would generate random points on the disk in polar coordinates with r = sqrt(random()); theta = 2 * pi * random().
Now instead of randomly sampling this inverse function we’re uniformly sampling it, and the nice thing about uniform sampling is that our results about how points are spread out in the limit of large N will behave as if we had randomly sampled it. This combination is the trick. Instead of random() we use (arange(0, num_pts, dtype=float) + 0.5)/num_pts, so that, say, if we want to sample 10 points they are r = 0.05, 0.15, 0.25, ... 0.95. We uniformly sample r to get equal-area spacing, and we use the sunflower increment to avoid awful “bars” of points in the output.
The changes that we need to make to dot the sphere with points merely involve switching out the polar coordinates for spherical coordinates. The radial coordinate of course doesn"t enter into this because we"re on a unit sphere. To keep things a little more consistent here, even though I was trained as a physicist I"ll use mathematicians" coordinates where 0 ≤ φ ≤ π is latitude coming down from the pole and 0 ≤ θ ≤ 2π is longitude. So the difference from above is that we are basically replacing the variable r with φ.
Our area element, which was r dr dθ, now becomes the not-much-more-complicated sin(φ) dφ dθ. So our joint density for uniform spacing is sin(φ)/4π. Integrating out θ, we find f(φ) = sin(φ)/2, thus F(φ) = (1 − cos(φ))/2. Inverting this we can see that a uniform random variable would look like acos(1 - 2 u), but we sample uniformly instead of randomly, so we instead use φk = acos(1 − 2 (k + 0.5)/N). And the rest of the algorithm is just projecting this onto the x, y, and z coordinates:
from numpy import pi, cos, sin, arccos, arange
import mpl_toolkits.mplot3d
import matplotlib.pyplot as pp
num_pts = 1000
indices = arange(0, num_pts, dtype=float) + 0.5
phi = arccos(1 - 2*indices/num_pts)
theta = pi * (1 + 5**0.5) * indices
x, y, z = cos(theta) * sin(phi), sin(theta) * sin(phi), cos(phi);
pp.figure().add_subplot(111, projection="3d").scatter(x, y, z);
pp.show()
Again for n=100 and n=1000 the results look like:
I wanted to give a shout out to Martin Roberts’s blog. Note that above I created an offset of my indices by adding 0.5 to each index. This was just visually appealing to me, but it turns out that the choice of offset matters a lot and is not constant over the interval and can mean getting as much as 8% better accuracy in packing if chosen correctly. There should also be a way to get his R2 sequence to cover a sphere and it would be interesting to see if this also produced a nice even covering, perhaps as-is but perhaps needing to be, say, taken from only a half of the unit square cut diagonally or so and stretched around to get a circle.
Those “bars” are formed by rational approximations to a number, and the best rational approximations to a number come from its continued fraction expression, z + 1/(n_1 + 1/(n_2 + 1/(n_3 + ...))) where z is an integer and n_1, n_2, n_3, ... is either a finite or infinite sequence of positive integers:
def continued_fraction(r):
while r != 0:
n = floor(r)
yield n
r = 1/(r - n)
Since the fraction part 1/(...) is always between zero and one, a large integer in the continued fraction allows for a particularly good rational approximation: “one divided by something between 100 and 101” is better than “one divided by something between 1 and 2.” The most irrational number is therefore the one which is 1 + 1/(1 + 1/(1 + ...)) and has no particularly good rational approximations; one can solve φ = 1 + 1/φ by multiplying through by φ to get the formula for the golden ratio.
For folks who are not so familiar with NumPy -- all of the functions are “vectorized,” so that sqrt(array) is the same as what other languages might write map(sqrt, array). So this is a component-by-component sqrt application. The same also holds for division by a scalar or addition with scalars -- those apply to all components in parallel.
The proof is simple once you know that this is the result. If you ask what"s the probability that z < Z < z + dz, this is the same as asking what"s the probability that z < F-1(U) < z + dz, apply F to all three expressions noting that it is a monotonically increasing function, hence F(z) < U < F(z + dz), expand the right hand side out to find F(z) + f(z) dz, and since U is uniform this probability is just f(z) dz as promised.
Python has no built-in encryption schemes, no. You also should take encrypted data storage serious; trivial encryption schemes that one developer understands to be insecure and a toy scheme may well be mistaken for a secure scheme by a less experienced developer. If you encrypt, encrypt properly.
You don’t need to do much work to implement a proper encryption scheme however. First of all, don’t re-invent the cryptography wheel, use a trusted cryptography library to handle this for you. For Python 3, that trusted library is cryptography.
I also recommend that encryption and decryption applies to bytes; encode text messages to bytes first; stringvalue.encode() encodes to UTF8, easily reverted again using bytesvalue.decode().
Last but not least, when encrypting and decrypting, we talk about keys, not passwords. A key should not be human memorable, it is something you store in a secret location but machine readable, whereas a password often can be human-readable and memorised. You can derive a key from a password, with a little care.
But for a web application or process running in a cluster without human attention to keep running it, you want to use a key. Passwords are for when only an end-user needs access to the specific information. Even then, you usually secure the application with a password, then exchange encrypted information using a key, perhaps one attached to the user account.
The cryptography library includes the Fernet recipe, a best-practices recipe for using cryptography. Fernet is an open standard,
with ready implementations in a wide range of programming languages and it packages AES CBC encryption for you with version information, a timestamp and an HMAC signature to prevent message tampering.
Fernet makes it very easy to encrypt and decrypt messages and keep you secure. It is the ideal method for encrypting data with a secret.
I recommend you use Fernet.generate_key() to generate a secure key. You can use a password too (next section), but a full 32-byte secret key (16 bytes to encrypt with, plus another 16 for the signature) is going to be more secure than most passwords you could think of.
The key that Fernet generates is a bytes object with URL and file safe base64 characters, so printable:
from cryptography.fernet import Fernet
key = Fernet.generate_key() # store in a secure location
print("Key:", key.decode())
To encrypt or decrypt messages, create a Fernet() instance with the given key, and call the Fernet.encrypt() or Fernet.decrypt(), both the plaintext message to encrypt and the encrypted token are bytes objects.
encrypt() and decrypt() functions would look like:
from cryptography.fernet import Fernet
def encrypt(message: bytes, key: bytes) -> bytes:
return Fernet(key).encrypt(message)
def decrypt(token: bytes, key: bytes) -> bytes:
return Fernet(key).decrypt(token)
Demo:
>>> key = Fernet.generate_key()
>>> print(key.decode())
GZWKEhHGNopxRdOHS4H4IyKhLQ8lwnyU7vRLrM3sebY=
>>> message = "John Doe"
>>> encrypt(message.encode(), key)
"gAAAAABciT3pFbbSihD_HZBZ8kqfAj94UhknamBuirZWKivWOukgKQ03qE2mcuvpuwCSuZ-X_Xkud0uWQLZ5e-aOwLC0Ccnepg=="
>>> token = _
>>> decrypt(token, key).decode()
"John Doe"
You can use a password instead of a secret key, provided you use a strong key derivation method. You do then have to include the salt and the HMAC iteration count in the message, so the encrypted value is not Fernet-compatible anymore without first separating salt, count and Fernet token:
import secrets
from base64 import urlsafe_b64encode as b64e, urlsafe_b64decode as b64d
from cryptography.fernet import Fernet
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
backend = default_backend()
iterations = 100_000
def _derive_key(password: bytes, salt: bytes, iterations: int = iterations) -> bytes:
"""Derive a secret key from a given password and salt"""
kdf = PBKDF2HMAC(
algorithm=hashes.SHA256(), length=32, salt=salt,
iterations=iterations, backend=backend)
return b64e(kdf.derive(password))
def password_encrypt(message: bytes, password: str, iterations: int = iterations) -> bytes:
salt = secrets.token_bytes(16)
key = _derive_key(password.encode(), salt, iterations)
return b64e(
b"%b%b%b" % (
salt,
iterations.to_bytes(4, "big"),
b64d(Fernet(key).encrypt(message)),
)
)
def password_decrypt(token: bytes, password: str) -> bytes:
decoded = b64d(token)
salt, iter, token = decoded[:16], decoded[16:20], b64e(decoded[20:])
iterations = int.from_bytes(iter, "big")
key = _derive_key(password.encode(), salt, iterations)
return Fernet(key).decrypt(token)
Demo:
>>> message = "John Doe"
>>> password = "mypass"
>>> password_encrypt(message.encode(), password)
b"9Ljs-w8IRM3XT1NDBbSBuQABhqCAAAAAAFyJdhiCPXms2vQHO7o81xZJn5r8_PAtro8Qpw48kdKrq4vt-551BCUbcErb_GyYRz8SVsu8hxTXvvKOn9QdewRGDfwx"
>>> token = _
>>> password_decrypt(token, password).decode()
"John Doe"
Including the salt in the output makes it possible to use a random salt value, which in turn ensures the encrypted output is guaranteed to be fully random regardless of password reuse or message repetition. Including the iteration count ensures that you can adjust for CPU performance increases over time without losing the ability to decrypt older messages.
A password alone can be as safe as a Fernet 32-byte random key, provided you generate a properly random password from a similar size pool. 32 bytes gives you 256 ^ 32 number of keys, so if you use an alphabet of 74 characters (26 upper, 26 lower, 10 digits and 12 possible symbols), then your password should be at least math.ceil(math.log(256 ** 32, 74)) == 42 characters long. However, a well-selected larger number of HMAC iterations can mitigate the lack of entropy somewhat as this makes it much more expensive for an attacker to brute force their way in.
Just know that choosing a shorter but still reasonably secure password won’t cripple this scheme, it just reduces the number of possible values a brute-force attacker would have to search through; make sure to pick a strong enough password for your security requirements.
An alternative is not to encrypt. Don"t be tempted to just use a low-security cipher, or a home-spun implementation of, say Vignere. There is no security in these approaches, but may give an inexperienced developer that is given the task to maintain your code in future the illusion of security, which is worse than no security at all.
If all you need is obscurity, just base64 the data; for URL-safe requirements, the base64.urlsafe_b64encode() function is fine. Don"t use a password here, just encode and you are done. At most, add some compression (like zlib):
import zlib
from base64 import urlsafe_b64encode as b64e, urlsafe_b64decode as b64d
def obscure(data: bytes) -> bytes:
return b64e(zlib.compress(data, 9))
def unobscure(obscured: bytes) -> bytes:
return zlib.decompress(b64d(obscured))
This turns b"Hello world!" into b"eNrzSM3JyVcozy_KSVEEAB0JBF4=".
If all you need is a way to make sure that the data can be trusted to be unaltered after having been sent to an untrusted client and received back, then you want to sign the data, you can use the hmac library for this with SHA1 (still considered secure for HMAC signing) or better:
import hmac
import hashlib
def sign(data: bytes, key: bytes, algorithm=hashlib.sha256) -> bytes:
assert len(key) >= algorithm().digest_size, (
"Key must be at least as long as the digest size of the "
"hashing algorithm"
)
return hmac.new(key, data, algorithm).digest()
def verify(signature: bytes, data: bytes, key: bytes, algorithm=hashlib.sha256) -> bytes:
expected = sign(data, key, algorithm)
return hmac.compare_digest(expected, signature)
Use this to sign data, then attach the signature with the data and send that to the client. When you receive the data back, split data and signature and verify. I"ve set the default algorithm to SHA256, so you"ll need a 32-byte key:
key = secrets.token_bytes(32)
You may want to look at the itsdangerous library, which packages this all up with serialisation and de-serialisation in various formats.
Fernet builds on AEC-CBC with a HMAC signature to ensure integrity of the encrypted data; a malicious attacker can"t feed your system nonsense data to keep your service busy running in circles with bad input, because the ciphertext is signed.
The Galois / Counter mode block cipher produces ciphertext and a tag to serve the same purpose, so can be used to serve the same purposes. The downside is that unlike Fernet there is no easy-to-use one-size-fits-all recipe to reuse on other platforms. AES-GCM also doesn"t use padding, so this encryption ciphertext matches the length of the input message (whereas Fernet / AES-CBC encrypts messages to blocks of fixed length, obscuring the message length somewhat).
AES256-GCM takes the usual 32 byte secret as a key:
key = secrets.token_bytes(32)
then use
import binascii, time
from base64 import urlsafe_b64encode as b64e, urlsafe_b64decode as b64d
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
from cryptography.exceptions import InvalidTag
backend = default_backend()
def aes_gcm_encrypt(message: bytes, key: bytes) -> bytes:
current_time = int(time.time()).to_bytes(8, "big")
algorithm = algorithms.AES(key)
iv = secrets.token_bytes(algorithm.block_size // 8)
cipher = Cipher(algorithm, modes.GCM(iv), backend=backend)
encryptor = cipher.encryptor()
encryptor.authenticate_additional_data(current_time)
ciphertext = encryptor.update(message) + encryptor.finalize()
return b64e(current_time + iv + ciphertext + encryptor.tag)
def aes_gcm_decrypt(token: bytes, key: bytes, ttl=None) -> bytes:
algorithm = algorithms.AES(key)
try:
data = b64d(token)
except (TypeError, binascii.Error):
raise InvalidToken
timestamp, iv, tag = data[:8], data[8:algorithm.block_size // 8 + 8], data[-16:]
if ttl is not None:
current_time = int(time.time())
time_encrypted, = int.from_bytes(data[:8], "big")
if time_encrypted + ttl < current_time or current_time + 60 < time_encrypted:
# too old or created well before our current time + 1 h to account for clock skew
raise InvalidToken
cipher = Cipher(algorithm, modes.GCM(iv, tag), backend=backend)
decryptor = cipher.decryptor()
decryptor.authenticate_additional_data(timestamp)
ciphertext = data[8 + len(iv):-16]
return decryptor.update(ciphertext) + decryptor.finalize()
I"ve included a timestamp to support the same time-to-live use-cases that Fernet supports.
This is the approach that All –Ü—ï V–∞–∏—ñ—Çy follows, albeit incorrectly. This is the cryptography version, but note that I include the IV in the ciphertext, it should not be stored as a global (reusing an IV weakens the security of the key, and storing it as a module global means it"ll be re-generated the next Python invocation, rendering all ciphertext undecryptable):
import secrets
from base64 import urlsafe_b64encode as b64e, urlsafe_b64decode as b64d
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
backend = default_backend()
def aes_cfb_encrypt(message, key):
algorithm = algorithms.AES(key)
iv = secrets.token_bytes(algorithm.block_size // 8)
cipher = Cipher(algorithm, modes.CFB(iv), backend=backend)
encryptor = cipher.encryptor()
ciphertext = encryptor.update(message) + encryptor.finalize()
return b64e(iv + ciphertext)
def aes_cfb_decrypt(ciphertext, key):
iv_ciphertext = b64d(ciphertext)
algorithm = algorithms.AES(key)
size = algorithm.block_size // 8
iv, encrypted = iv_ciphertext[:size], iv_ciphertext[size:]
cipher = Cipher(algorithm, modes.CFB(iv), backend=backend)
decryptor = cipher.decryptor()
return decryptor.update(encrypted) + decryptor.finalize()
This lacks the added armoring of an HMAC signature and there is no timestamp; you’d have to add those yourself.
The above also illustrates how easy it is to combine basic cryptography building blocks incorrectly; All Іѕ Vаиітy‘s incorrect handling of the IV value can lead to a data breach or all encrypted messages being unreadable because the IV is lost. Using Fernet instead protects you from such mistakes.
If you previously implemented AES ECB encryption and need to still support this in Python 3, you can do so still with cryptography too. The same caveats apply, ECB is not secure enough for real-life applications. Re-implementing that answer for Python 3, adding automatic handling of padding:
from base64 import urlsafe_b64encode as b64e, urlsafe_b64decode as b64d
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.primitives import padding
from cryptography.hazmat.backends import default_backend
backend = default_backend()
def aes_ecb_encrypt(message, key):
cipher = Cipher(algorithms.AES(key), modes.ECB(), backend=backend)
encryptor = cipher.encryptor()
padder = padding.PKCS7(cipher.algorithm.block_size).padder()
padded = padder.update(msg_text.encode()) + padder.finalize()
return b64e(encryptor.update(padded) + encryptor.finalize())
def aes_ecb_decrypt(ciphertext, key):
cipher = Cipher(algorithms.AES(key), modes.ECB(), backend=backend)
decryptor = cipher.decryptor()
unpadder = padding.PKCS7(cipher.algorithm.block_size).unpadder()
padded = decryptor.update(b64d(ciphertext)) + decryptor.finalize()
return unpadder.update(padded) + unpadder.finalize()
Again, this lacks the HMAC signature, and you shouldn’t use ECB anyway. The above is there merely to illustrate that cryptography can handle the common cryptographic building blocks, even the ones you shouldn’t actually use.
import matplotlib.pyplot as plt
circle1 = plt.Circle((0, 0), 0.2, color="r")
plt.gca().add_patch(circle1)
A quick condensed version of the accepted answer, to quickly plug a circle into an existing plot. Refer to the accepted answer and other answers to understand the details.
By the way:
gca() means Get Current AxisThere is a picture show all markers" name and description, i hope it will help you.
import matplotlib.pylab as plt
markers = [".",",","o","v","^","<",">","1","2","3","4","8","s","p","P","*","h","H","+","x","X","D","d","|","_"]
descriptions = ["point", "pixel", "circle", "triangle_down", "triangle_up","triangle_left",
"triangle_right", "tri_down", "tri_up", "tri_left", "tri_right", "octagon",
"square", "pentagon", "plus (filled)","star", "hexagon1", "hexagon2", "plus",
"x", "x (filled)","diamond", "thin_diamond", "vline", "hline"]
x=[]
y=[]
for i in range(5):
for j in range(5):
x.append(i)
y.append(j)
plt.figure(figsize=(8, 8))
for i,j,m,l in zip(x,y,markers,descriptions):
plt.scatter(i,j,marker=m)
plt.text(i-0.15,j+0.15,s=m+" : "+l)
plt.axis([-0.1,4.8,-0.1,4.5])
plt.axis("off")
plt.tight_layout()
plt.show()
By definition, isdecimal() ⊆ isdigit() ⊆ isnumeric(). That is, if a string is decimal, then it"ll also be digit and numeric.
Therefore, given a string s and test it with those three methods, there"ll only be 4 types of results.
+-------------+-----------+-------------+----------------------------------+
| isdecimal() | isdigit() | isnumeric() | Example |
+-------------+-----------+-------------+----------------------------------+
| True | True | True | "038", "੦੩੮", "０３８" |
| False | True | True | "‚Å∞¬≥‚Å∏", "üÑÄ‚íä‚íè", "‚왂뢂ëß" |
| False | False | True | "↉⅛⅘", "ⅠⅢⅧ", "⑩⑬㊿", "壹貳參" |
| False | False | False | "abc", "38.0", "-38" |
+-------------+-----------+-------------+----------------------------------+
1. Some examples of characters isdecimal()==True
(thus isdigit()==True and isnumeric()==True)
"0123456789" DIGIT ZERO~NINE
"٠١٢٣٤٥٦٧٨٩" ARABIC-INDIC DIGIT ZERO~NINE
"०१२३४५६७८९" DEVANAGARI DIGIT ZERO~NINE
"০১২৩৪৫৬৭৮৯" BENGALI DIGIT ZERO~NINE
"੦੧੨੩੪੫੬੭੮੯" GURMUKHI DIGIT ZERO~NINE
"૦૧૨૩૪૫૬૭૮૯" GUJARATI DIGIT ZERO~NINE
"୦୧୨୩୪୫୬୭୮୯" ORIYA DIGIT ZERO~NINE
"௦௧௨௩௪௫௬௭௮௯" TAMIL DIGIT ZERO~NINE
"౦౧౨౩౪౫౬౭౮౯" TELUGU DIGIT ZERO~NINE
"೦೧೨೩೪೫೬೭೮೯" KANNADA DIGIT ZERO~NINE
"൦൧൨൩൪൫൬൭൮൯" MALAYALAM DIGIT ZERO~NINE
"๐๑๒๓๔๕๖๗๘๙" THAI DIGIT ZERO~NINE
"໐໑໒໓໔໕໖໗໘໙" LAO DIGIT ZERO~NINE
"༠༡༢༣༤༥༦༧༨༩" TIBETAN DIGIT ZERO~NINE
"·ÅÄ·ÅÅ·ÅÇ·ÅÉ·ÅÑ·ÅÖ·ÅÜ·Åá·Åà·Åâ" MYANMAR DIGIT ZERO~NINE
"០១២៣៤៥៦៧៨៩" KHMER DIGIT ZERO~NINE
"０１２３４５６７８９" FULLWIDTH DIGIT ZERO~NINE
"ùüéùüèùüêùüëùüíùüìùüîùüïùüñùüó" MATHEMATICAL BOLD DIGIT ZERO~NINE
"ùüòùüôùüöùüõùüúùüùùüûùüüùü†ùü°" MATHEMATICAL DOUBLE-STRUCK DIGIT ZERO~NINE
"ùü¢ùü£ùü§ùü•ùü¶ùüßùü®ùü©ùü™ùü´" MATHEMATICAL SANS-SERIF DIGIT ZERO~NINE
"ùü¨ùü≠ùüÆùüØùü∞ùü±ùü≤ùü≥ùü¥ùüµ" MATHEMATICAL SANS-SERIF BOLD DIGIT ZERO~NINE
"ùü∂ùü∑ùü∏ùüπùü∫ùüªùüºùüΩùüæùüø" MATHEMATICAL MONOSPACE DIGIT ZERO~NINE
2. Some examples of characters isdecimal()==False but isdigit()==True
(thus isnumeric()==True)
"⁰¹²³⁴⁵⁶⁷⁸⁹" SUPERSCRIPT ZERO~NINE
"‚ÇÄ‚ÇÅ‚ÇÇ‚ÇÉ‚ÇÑ‚ÇÖ‚ÇÜ‚Çá‚Çà‚Çâ" SUBSCRIPT ZERO~NINE
"üÑÄ‚íà‚íâ‚íä‚íã‚íå‚íç‚íé‚íè‚íê" DIGIT ZERO~NINE FULL STOP
"üÑÅüÑÇüÑÉüÑÑüÑÖüÑÜüÑáüÑàüÑâüÑä" DIGIT ZERO~NINE COMMA
"⓪①②③④⑤⑥⑦⑧⑨" CIRCLED DIGIT ZERO~NINE
"⓿❶❷❸❹❺❻❼❽❾" NEGATIVE CIRCLED DIGIT ZERO~NINE
"⑴⑵⑶⑷⑸⑹⑺⑻⑼" PARENTHESIZED DIGIT ONE~NINE
"‚ûÄ‚ûÅ‚ûÇ‚ûÉ‚ûÑ‚ûÖ‚ûÜ‚ûá‚ûà" DINGBAT CIRCLED SANS-SERIF DIGIT ONE~NINE
"⓵⓶⓷⓸⓹⓺⓻⓼⓽" DOUBLE CIRCLED DIGIT ONE~NINE
"‚ûä‚ûã‚ûå‚ûç‚ûé‚ûè‚ûê‚ûë‚ûí" DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT ONE~NINE
"፩፪፫፬፭፮፯፰፱" ETHIOPIC DIGIT ONE~NINE
3. Some examples of characters isdecimal()==False and isdigit()==False but isnumeric()==True
"½⅓¼⅕⅙⅐⅛⅑⅒⅔¾⅖⅗⅘⅚⅜⅝⅞⅟↉" VULGAR FRACTION
"৴৵৶৷৸৹" BENGALI CURRENCY NUMERATOR
"௰௱௲" TAMIL NUMBER TEN, ONE HUNDRED, ONE THOUSAND
"౸౹౺౻౼౽౾" TELUGU FRACTION DIGIT
"൰൱൲൳൴൵" MALAYALAM NUMBER, MALAYALAM FRACTION
"༳༪༫༬༭༮༯༰༱༲" TIBETAN DIGIT HALF ZERO~NINE
"፲፳፴፵፶፷፸፹፺፻፼" ETHIOPIC NUMBER TEN~NINETY, HUNDRED, TEN THOUSAND
"៰៱៲៳៴៵៶៷៸៹" KHMER SYMBOL LEK ATTAK
"ⅠⅡⅢⅣⅤⅥⅦⅧⅨⅩⅪⅫⅬⅭⅮⅯ" ROMAN NUMERAL
"ⅰⅱⅲⅳⅴⅵⅶⅷⅸⅹⅺⅻⅼⅽⅾⅿ" SMALL ROMAN NUMERAL
"‚ÜÄ‚ÜÅ‚ÜÇ‚ÜÖ‚ÜÜ" ROMAN NUMERAL
"⑩⑪⑫⑬⑭⑮⑯⑰⑱⑲⑳㉑㉒㉓㉔㉕㉖㉗㉘㉙㉚㉛㉜㉝㉞㉟㊱㊲㊳㊴㊵㊶㊷㊸㊹㊺㊻㊼㊽㊾㊿" CIRCLED NUMBER TEN~FIFTY
"„âà„ââ„âä„âã„âå„âç„âé„âè" CIRCLED NUMBER TEN~EIGHTY ON BLACK SQUARE
"⑽⑾⑿⒀⒁⒂⒃⒄⒅⒆⒇" PARENTHESIZED NUMBER TEN~TWENTY
"‚íë‚íí‚íì‚íî‚íï‚íñ‚íó‚íò‚íô‚íö‚íõ" NUMBER TEN~TWENTY FULL STOP
"⓫⓬⓭⓮⓯⓰⓱⓲⓳⓴" NEGATIVE CIRCLED NUMBER ELEVEN
"‚ìæ‚ûâ‚ùø‚ûì" various styles of CIRCLED NUMBER TEN
"üÑå" DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT ZERO
"„Äá" IDEOGRAPHIC NUMBER ZERO
"〡〢〣〤〥〦〧〨〩〸〹〺" HANGZHOU NUMERAL ONE~TEN, TWENTY, THIRTY
"„Üí„Üì„Üî„Üï" IDEOGRAPHIC ANNOTATION ONE~FOUR MARK
"㈠㈡㈢㈣㈤㈥㈦㈧㈨㈩" PARENTHESIZED IDEOGRAPH ONE~TEN
"㊀㊁㊂㊃㊄㊅㊆㊇㊈㊉" CIRCLED IDEOGRAPH ONE~TEN
"一二三四五六七八九十壹貳參肆伍陸柒捌玖拾零百千萬億兆弐貮贰㒃㭍漆什㐅陌阡佰仟万亿幺兩㠪亖卄卅卌廾廿" CJK UNIFIED IDEOGRAPH
"參拾兩零六陸什" CJK COMPATIBILITY IDEOGRAPH
"êÑáêÑàêÑâêÑäêÑãêÑåêÑçêÑéêÑèêÑêêÑëêÑíêÑìêÑîêÑïêÑñêÑóêÑò" AEGEAN NUMBER ONE~NINE, TEN~NINETY
"êÑôêÑöêÑõêÑúêÑùêÑûêÑüêцêÑ°êÑ¢êÑ£êѧêÑ•êѶêÑßêÑ®êÑ©êÑ™" AEGEAN NUMBER ONE~NINE HUNDRED, ONE~NINE THOUSAND
"êѨêÑ≠êÑÆêÑØêÑ∞êѱêÑ≤êÑ≥" AEGEAN NUMBER TEN~NINETY THOUSAND
"êÖÄêÖÅêÖÇêÖÉêÖÜêÖáêÖàêÖâêÖäêÖãêÖåêÖçêÖéêÖèêÖêêÖëêÖíêÖìêÖîêÖïêÖñêÖóêÖòêÖôêÖöêÖõêÖúêÖùêÖûêÖüêÖ†êÖ°êÖ¢êÖ£êÖ§êÖ•êÖ¶êÖßêÖ®êÖ©êÖ™êÖ´êÖ¨êÖ≠êÖÆêÖØêÖ∞êÖ±êÖ≤êÖ≥êÖ¥" GREEK ACROPHONIC ATTIC
"ùç†ùç°ùç¢ùç£ùç§ùç•ùç¶ùçßùç®" COUNTING ROD UNIT DIGIT ONE~NINE
"ùç©ùç™ùç´ùç¨ùç≠ùçÆùçØùç∞ùç±" COUNTING ROD TENS DIGIT ONE~NINE
@Jan Kuiken"s answer is certainly well-thought and thorough, but there are some caveats:
A much simpler approach is to annotate the last point of each plot. The point can also be circled, for emphasis. This can be accomplished with one extra line:
from matplotlib import pyplot as plt
for i, (x, y) in enumerate(samples):
plt.plot(x, y)
plt.text(x[-1], y[-1], "sample {i}".format(i=i))
A variant would be to use ax.annotate.
If you are not into long explanations, see Paolo Bergantino’s answer.
To understand decorators, you must first understand that functions are objects in Python. This has important consequences. Let’s see why with a simple example :
def shout(word="yes"):
return word.capitalize()+"!"
print(shout())
# outputs : "Yes!"
# As an object, you can assign the function to a variable like any other object
scream = shout
# Notice we don"t use parentheses: we are not calling the function,
# we are putting the function "shout" into the variable "scream".
# It means you can then call "shout" from "scream":
print(scream())
# outputs : "Yes!"
# More than that, it means you can remove the old name "shout",
# and the function will still be accessible from "scream"
del shout
try:
print(shout())
except NameError as e:
print(e)
#outputs: "name "shout" is not defined"
print(scream())
# outputs: "Yes!"
Keep this in mind. We’ll circle back to it shortly.
Another interesting property of Python functions is they can be defined inside another function!
def talk():
# You can define a function on the fly in "talk" ...
def whisper(word="yes"):
return word.lower()+"..."
# ... and use it right away!
print(whisper())
# You call "talk", that defines "whisper" EVERY TIME you call it, then
# "whisper" is called in "talk".
talk()
# outputs:
# "yes..."
# But "whisper" DOES NOT EXIST outside "talk":
try:
print(whisper())
except NameError as e:
print(e)
#outputs : "name "whisper" is not defined"*
#Python"s functions are objects
Okay, still here? Now the fun part...
You’ve seen that functions are objects. Therefore, functions:
That means that a function can return another function.
def getTalk(kind="shout"):
# We define functions on the fly
def shout(word="yes"):
return word.capitalize()+"!"
def whisper(word="yes") :
return word.lower()+"..."
# Then we return one of them
if kind == "shout":
# We don"t use "()", we are not calling the function,
# we are returning the function object
return shout
else:
return whisper
# How do you use this strange beast?
# Get the function and assign it to a variable
talk = getTalk()
# You can see that "talk" is here a function object:
print(talk)
#outputs : <function shout at 0xb7ea817c>
# The object is the one returned by the function:
print(talk())
#outputs : Yes!
# And you can even use it directly if you feel wild:
print(getTalk("whisper")())
#outputs : yes...
There’s more!
If you can return a function, you can pass one as a parameter:
def doSomethingBefore(func):
print("I do something before then I call the function you gave me")
print(func())
doSomethingBefore(scream)
#outputs:
#I do something before then I call the function you gave me
#Yes!
Well, you just have everything needed to understand decorators. You see, decorators are “wrappers”, which means that they let you execute code before and after the function they decorate without modifying the function itself.
How you’d do it manually:
# A decorator is a function that expects ANOTHER function as parameter
def my_shiny_new_decorator(a_function_to_decorate):
# Inside, the decorator defines a function on the fly: the wrapper.
# This function is going to be wrapped around the original function
# so it can execute code before and after it.
def the_wrapper_around_the_original_function():
# Put here the code you want to be executed BEFORE the original function is called
print("Before the function runs")
# Call the function here (using parentheses)
a_function_to_decorate()
# Put here the code you want to be executed AFTER the original function is called
print("After the function runs")
# At this point, "a_function_to_decorate" HAS NEVER BEEN EXECUTED.
# We return the wrapper function we have just created.
# The wrapper contains the function and the code to execute before and after. It’s ready to use!
return the_wrapper_around_the_original_function
# Now imagine you create a function you don"t want to ever touch again.
def a_stand_alone_function():
print("I am a stand alone function, don"t you dare modify me")
a_stand_alone_function()
#outputs: I am a stand alone function, don"t you dare modify me
# Well, you can decorate it to extend its behavior.
# Just pass it to the decorator, it will wrap it dynamically in
# any code you want and return you a new function ready to be used:
a_stand_alone_function_decorated = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function_decorated()
#outputs:
#Before the function runs
#I am a stand alone function, don"t you dare modify me
#After the function runs
Now, you probably want that every time you call a_stand_alone_function, a_stand_alone_function_decorated is called instead. That’s easy, just overwrite a_stand_alone_function with the function returned by my_shiny_new_decorator:
a_stand_alone_function = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function()
#outputs:
#Before the function runs
#I am a stand alone function, don"t you dare modify me
#After the function runs
# That’s EXACTLY what decorators do!
The previous example, using the decorator syntax:
@my_shiny_new_decorator
def another_stand_alone_function():
print("Leave me alone")
another_stand_alone_function()
#outputs:
#Before the function runs
#Leave me alone
#After the function runs
Yes, that’s all, it’s that simple. @decorator is just a shortcut to:
another_stand_alone_function = my_shiny_new_decorator(another_stand_alone_function)
Decorators are just a pythonic variant of the decorator design pattern. There are several classic design patterns embedded in Python to ease development (like iterators).
Of course, you can accumulate decorators:
def bread(func):
def wrapper():
print("</"""""">")
func()
print("<\______/>")
return wrapper
def ingredients(func):
def wrapper():
print("#tomatoes#")
func()
print("~salad~")
return wrapper
def sandwich(food="--ham--"):
print(food)
sandwich()
#outputs: --ham--
sandwich = bread(ingredients(sandwich))
sandwich()
#outputs:
#</"""""">
# #tomatoes#
# --ham--
# ~salad~
#<\______/>
Using the Python decorator syntax:
@bread
@ingredients
def sandwich(food="--ham--"):
print(food)
sandwich()
#outputs:
#</"""""">
# #tomatoes#
# --ham--
# ~salad~
#<\______/>
The order you set the decorators MATTERS:
@ingredients
@bread
def strange_sandwich(food="--ham--"):
print(food)
strange_sandwich()
#outputs:
##tomatoes#
#</"""""">
# --ham--
#<\______/>
# ~salad~
As a conclusion, you can easily see how to answer the question:
# The decorator to make it bold
def makebold(fn):
# The new function the decorator returns
def wrapper():
# Insertion of some code before and after
return "<b>" + fn() + "</b>"
return wrapper
# The decorator to make it italic
def makeitalic(fn):
# The new function the decorator returns
def wrapper():
# Insertion of some code before and after
return "<i>" + fn() + "</i>"
return wrapper
@makebold
@makeitalic
def say():
return "hello"
print(say())
#outputs: <b><i>hello</i></b>
# This is the exact equivalent to
def say():
return "hello"
say = makebold(makeitalic(say))
print(say())
#outputs: <b><i>hello</i></b>
You can now just leave happy, or burn your brain a little bit more and see advanced uses of decorators.
# It’s not black magic, you just have to let the wrapper
# pass the argument:
def a_decorator_passing_arguments(function_to_decorate):
def a_wrapper_accepting_arguments(arg1, arg2):
print("I got args! Look: {0}, {1}".format(arg1, arg2))
function_to_decorate(arg1, arg2)
return a_wrapper_accepting_arguments
# Since when you are calling the function returned by the decorator, you are
# calling the wrapper, passing arguments to the wrapper will let it pass them to
# the decorated function
@a_decorator_passing_arguments
def print_full_name(first_name, last_name):
print("My name is {0} {1}".format(first_name, last_name))
print_full_name("Peter", "Venkman")
# outputs:
#I got args! Look: Peter Venkman
#My name is Peter Venkman
One nifty thing about Python is that methods and functions are really the same. The only difference is that methods expect that their first argument is a reference to the current object (self).
That means you can build a decorator for methods the same way! Just remember to take self into consideration:
def method_friendly_decorator(method_to_decorate):
def wrapper(self, lie):
lie = lie - 3 # very friendly, decrease age even more :-)
return method_to_decorate(self, lie)
return wrapper
class Lucy(object):
def __init__(self):
self.age = 32
@method_friendly_decorator
def sayYourAge(self, lie):
print("I am {0}, what did you think?".format(self.age + lie))
l = Lucy()
l.sayYourAge(-3)
#outputs: I am 26, what did you think?
If you’re making general-purpose decorator--one you’ll apply to any function or method, no matter its arguments--then just use *args, **kwargs:
def a_decorator_passing_arbitrary_arguments(function_to_decorate):
# The wrapper accepts any arguments
def a_wrapper_accepting_arbitrary_arguments(*args, **kwargs):
print("Do I have args?:")
print(args)
print(kwargs)
# Then you unpack the arguments, here *args, **kwargs
# If you are not familiar with unpacking, check:
# http://www.saltycrane.com/blog/2008/01/how-to-use-args-and-kwargs-in-python/
function_to_decorate(*args, **kwargs)
return a_wrapper_accepting_arbitrary_arguments
@a_decorator_passing_arbitrary_arguments
def function_with_no_argument():
print("Python is cool, no argument here.")
function_with_no_argument()
#outputs
#Do I have args?:
#()
#{}
#Python is cool, no argument here.
@a_decorator_passing_arbitrary_arguments
def function_with_arguments(a, b, c):
print(a, b, c)
function_with_arguments(1,2,3)
#outputs
#Do I have args?:
#(1, 2, 3)
#{}
#1 2 3
@a_decorator_passing_arbitrary_arguments
def function_with_named_arguments(a, b, c, platypus="Why not ?"):
print("Do {0}, {1} and {2} like platypus? {3}".format(a, b, c, platypus))
function_with_named_arguments("Bill", "Linus", "Steve", platypus="Indeed!")
#outputs
#Do I have args ? :
#("Bill", "Linus", "Steve")
#{"platypus": "Indeed!"}
#Do Bill, Linus and Steve like platypus? Indeed!
class Mary(object):
def __init__(self):
self.age = 31
@a_decorator_passing_arbitrary_arguments
def sayYourAge(self, lie=-3): # You can now add a default value
print("I am {0}, what did you think?".format(self.age + lie))
m = Mary()
m.sayYourAge()
#outputs
# Do I have args?:
#(<__main__.Mary object at 0xb7d303ac>,)
#{}
#I am 28, what did you think?
Great, now what would you say about passing arguments to the decorator itself?
This can get somewhat twisted, since a decorator must accept a function as an argument. Therefore, you cannot pass the decorated function’s arguments directly to the decorator.
Before rushing to the solution, let’s write a little reminder:
# Decorators are ORDINARY functions
def my_decorator(func):
print("I am an ordinary function")
def wrapper():
print("I am function returned by the decorator")
func()
return wrapper
# Therefore, you can call it without any "@"
def lazy_function():
print("zzzzzzzz")
decorated_function = my_decorator(lazy_function)
#outputs: I am an ordinary function
# It outputs "I am an ordinary function", because that’s just what you do:
# calling a function. Nothing magic.
@my_decorator
def lazy_function():
print("zzzzzzzz")
#outputs: I am an ordinary function
It’s exactly the same. "my_decorator" is called. So when you @my_decorator, you are telling Python to call the function "labelled by the variable "my_decorator"".
This is important! The label you give can point directly to the decorator—or not.
Let’s get evil. ☺
def decorator_maker():
print("I make decorators! I am executed only once: "
"when you make me create a decorator.")
def my_decorator(func):
print("I am a decorator! I am executed only when you decorate a function.")
def wrapped():
print("I am the wrapper around the decorated function. "
"I am called when you call the decorated function. "
"As the wrapper, I return the RESULT of the decorated function.")
return func()
print("As the decorator, I return the wrapped function.")
return wrapped
print("As a decorator maker, I return a decorator")
return my_decorator
# Let’s create a decorator. It’s just a new function after all.
new_decorator = decorator_maker()
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
# Then we decorate the function
def decorated_function():
print("I am the decorated function.")
decorated_function = new_decorator(decorated_function)
#outputs:
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function
# Let’s call the function:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.
No surprise here.
Let’s do EXACTLY the same thing, but skip all the pesky intermediate variables:
def decorated_function():
print("I am the decorated function.")
decorated_function = decorator_maker()(decorated_function)
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.
# Finally:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.
Let’s make it even shorter:
@decorator_maker()
def decorated_function():
print("I am the decorated function.")
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.
#Eventually:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.
Hey, did you see that? We used a function call with the "@" syntax! :-)
So, back to decorators with arguments. If we can use functions to generate the decorator on the fly, we can pass arguments to that function, right?
def decorator_maker_with_arguments(decorator_arg1, decorator_arg2):
print("I make decorators! And I accept arguments: {0}, {1}".format(decorator_arg1, decorator_arg2))
def my_decorator(func):
# The ability to pass arguments here is a gift from closures.
# If you are not comfortable with closures, you can assume it’s ok,
# or read: https://stackoverflow.com/questions/13857/can-you-explain-closures-as-they-relate-to-python
print("I am the decorator. Somehow you passed me arguments: {0}, {1}".format(decorator_arg1, decorator_arg2))
# Don"t confuse decorator arguments and function arguments!
def wrapped(function_arg1, function_arg2) :
print("I am the wrapper around the decorated function.
"
"I can access all the variables
"
" - from the decorator: {0} {1}
"
" - from the function call: {2} {3}
"
"Then I can pass them to the decorated function"
.format(decorator_arg1, decorator_arg2,
function_arg1, function_arg2))
return func(function_arg1, function_arg2)
return wrapped
return my_decorator
@decorator_maker_with_arguments("Leonard", "Sheldon")
def decorated_function_with_arguments(function_arg1, function_arg2):
print("I am the decorated function and only knows about my arguments: {0}"
" {1}".format(function_arg1, function_arg2))
decorated_function_with_arguments("Rajesh", "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Sheldon
#I am the decorator. Somehow you passed me arguments: Leonard Sheldon
#I am the wrapper around the decorated function.
#I can access all the variables
# - from the decorator: Leonard Sheldon
# - from the function call: Rajesh Howard
#Then I can pass them to the decorated function
#I am the decorated function and only knows about my arguments: Rajesh Howard
Here it is: a decorator with arguments. Arguments can be set as variable:
c1 = "Penny"
c2 = "Leslie"
@decorator_maker_with_arguments("Leonard", c1)
def decorated_function_with_arguments(function_arg1, function_arg2):
print("I am the decorated function and only knows about my arguments:"
" {0} {1}".format(function_arg1, function_arg2))
decorated_function_with_arguments(c2, "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Penny
#I am the decorator. Somehow you passed me arguments: Leonard Penny
#I am the wrapper around the decorated function.
#I can access all the variables
# - from the decorator: Leonard Penny
# - from the function call: Leslie Howard
#Then I can pass them to the decorated function
#I am the decorated function and only know about my arguments: Leslie Howard
As you can see, you can pass arguments to the decorator like any function using this trick. You can even use *args, **kwargs if you wish. But remember decorators are called only once. Just when Python imports the script. You can"t dynamically set the arguments afterwards. When you do "import x", the function is already decorated, so you can"t
change anything.
Okay, as a bonus, I"ll give you a snippet to make any decorator accept generically any argument. After all, in order to accept arguments, we created our decorator using another function.
We wrapped the decorator.
Anything else we saw recently that wrapped function?
Oh yes, decorators!
Let’s have some fun and write a decorator for the decorators:
def decorator_with_args(decorator_to_enhance):
"""
This function is supposed to be used as a decorator.
It must decorate an other function, that is intended to be used as a decorator.
Take a cup of coffee.
It will allow any decorator to accept an arbitrary number of arguments,
saving you the headache to remember how to do that every time.
"""
# We use the same trick we did to pass arguments
def decorator_maker(*args, **kwargs):
# We create on the fly a decorator that accepts only a function
# but keeps the passed arguments from the maker.
def decorator_wrapper(func):
# We return the result of the original decorator, which, after all,
# IS JUST AN ORDINARY FUNCTION (which returns a function).
# Only pitfall: the decorator must have this specific signature or it won"t work:
return decorator_to_enhance(func, *args, **kwargs)
return decorator_wrapper
return decorator_maker
It can be used as follows:
# You create the function you will use as a decorator. And stick a decorator on it :-)
# Don"t forget, the signature is "decorator(func, *args, **kwargs)"
@decorator_with_args
def decorated_decorator(func, *args, **kwargs):
def wrapper(function_arg1, function_arg2):
print("Decorated with {0} {1}".format(args, kwargs))
return func(function_arg1, function_arg2)
return wrapper
# Then you decorate the functions you wish with your brand new decorated decorator.
@decorated_decorator(42, 404, 1024)
def decorated_function(function_arg1, function_arg2):
print("Hello {0} {1}".format(function_arg1, function_arg2))
decorated_function("Universe and", "everything")
#outputs:
#Decorated with (42, 404, 1024) {}
#Hello Universe and everything
# Whoooot!
I know, the last time you had this feeling, it was after listening a guy saying: "before understanding recursion, you must first understand recursion". But now, don"t you feel good about mastering this?
The functools module was introduced in Python 2.5. It includes the function functools.wraps(), which copies the name, module, and docstring of the decorated function to its wrapper.
(Fun fact: functools.wraps() is a decorator! ‚ò∫)
# For debugging, the stacktrace prints you the function __name__
def foo():
print("foo")
print(foo.__name__)
#outputs: foo
# With a decorator, it gets messy
def bar(func):
def wrapper():
print("bar")
return func()
return wrapper
@bar
def foo():
print("foo")
print(foo.__name__)
#outputs: wrapper
# "functools" can help for that
import functools
def bar(func):
# We say that "wrapper", is wrapping "func"
# and the magic begins
@functools.wraps(func)
def wrapper():
print("bar")
return func()
return wrapper
@bar
def foo():
print("foo")
print(foo.__name__)
#outputs: foo
Now the big question: What can I use decorators for?
Seem cool and powerful, but a practical example would be great. Well, there are 1000 possibilities. Classic uses are extending a function behavior from an external lib (you can"t modify it), or for debugging (you don"t want to modify it because it’s temporary).
You can use them to extend several functions in a DRY’s way, like so:
def benchmark(func):
"""
A decorator that prints the time a function takes
to execute.
"""
import time
def wrapper(*args, **kwargs):
t = time.clock()
res = func(*args, **kwargs)
print("{0} {1}".format(func.__name__, time.clock()-t))
return res
return wrapper
def logging(func):
"""
A decorator that logs the activity of the script.
(it actually just prints it, but it could be logging!)
"""
def wrapper(*args, **kwargs):
res = func(*args, **kwargs)
print("{0} {1} {2}".format(func.__name__, args, kwargs))
return res
return wrapper
def counter(func):
"""
A decorator that counts and prints the number of times a function has been executed
"""
def wrapper(*args, **kwargs):
wrapper.count = wrapper.count + 1
res = func(*args, **kwargs)
print("{0} has been used: {1}x".format(func.__name__, wrapper.count))
return res
wrapper.count = 0
return wrapper
@counter
@benchmark
@logging
def reverse_string(string):
return str(reversed(string))
print(reverse_string("Able was I ere I saw Elba"))
print(reverse_string("A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!"))
#outputs:
#reverse_string ("Able was I ere I saw Elba",) {}
#wrapper 0.0
#wrapper has been used: 1x
#ablE was I ere I saw elbA
#reverse_string ("A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!",) {}
#wrapper 0.0
#wrapper has been used: 2x
#!amanaP :lanac a ,noep a ,stah eros ,raj a ,hsac ,oloR a ,tur a ,mapS ,snip ,eperc a ,)lemac a ro( niaga gab ananab a ,gat a ,nat a ,gab ananab a ,gag a ,inoracam ,elacrep ,epins ,spam ,arutaroloc a ,shajar ,soreh ,atsap ,eonac a ,nalp a ,nam A
Of course the good thing with decorators is that you can use them right away on almost anything without rewriting. DRY, I said:
@counter
@benchmark
@logging
def get_random_futurama_quote():
from urllib import urlopen
result = urlopen("http://subfusion.net/cgi-bin/quote.pl?quote=futurama").read()
try:
value = result.split("<br><b><hr><br>")[1].split("<br><br><hr>")[0]
return value.strip()
except:
return "No, I"m ... doesn"t!"
print(get_random_futurama_quote())
print(get_random_futurama_quote())
#outputs:
#get_random_futurama_quote () {}
#wrapper 0.02
#wrapper has been used: 1x
#The laws of science be a harsh mistress.
#get_random_futurama_quote () {}
#wrapper 0.01
#wrapper has been used: 2x
#Curse you, merciful Poseidon!
Python itself provides several decorators: property, staticmethod, etc.
This really is a large playground.
Specify the keyword args linestyle and/or marker in your call to plot.
For example, using a dashed line and blue circle markers:
plt.plot(range(10), linestyle="--", marker="o", color="b")
A shortcut call for the same thing:
plt.plot(range(10), "--bo")
Here is a list of the possible line and marker styles:
================ ===============================
character description
================ ===============================
- solid line style
-- dashed line style
-. dash-dot line style
: dotted line style
. point marker
, pixel marker
o circle marker
v triangle_down marker
^ triangle_up marker
< triangle_left marker
> triangle_right marker
1 tri_down marker
2 tri_up marker
3 tri_left marker
4 tri_right marker
s square marker
p pentagon marker
* star marker
h hexagon1 marker
H hexagon2 marker
+ plus marker
x x marker
D diamond marker
d thin_diamond marker
| vline marker
_ hline marker
================ ===============================
edit: with an example of marking an arbitrary subset of points, as requested in the comments:
import numpy as np
import matplotlib.pyplot as plt
xs = np.linspace(-np.pi, np.pi, 30)
ys = np.sin(xs)
markers_on = [12, 17, 18, 19]
plt.plot(xs, ys, "-gD", markevery=markers_on)
plt.show()
This last example using the markevery kwarg is possible in since 1.4+, due to the merge of this feature branch. If you are stuck on an older version of matplotlib, you can still achieve the result by overlaying a scatterplot on the line plot. See the edit history for more details.
First find the difference between the start point and the end point (here, this is more of a directed line segment, not a "line", since lines extend infinitely and don"t start at a particular point).
deltaY = P2_y - P1_y
deltaX = P2_x - P1_x
Then calculate the angle (which runs from the positive X axis at P1 to the positive Y axis at P1).
angleInDegrees = arctan(deltaY / deltaX) * 180 / PI
But arctan may not be ideal, because dividing the differences this way will erase the distinction needed to distinguish which quadrant the angle is in (see below). Use the following instead if your language includes an atan2 function:
angleInDegrees = atan2(deltaY, deltaX) * 180 / PI
EDIT (Feb. 22, 2017): In general, however, calling atan2(deltaY,deltaX) just to get the proper angle for cos and sin may be inelegant. In those cases, you can often do the following instead:
(deltaX, deltaY) as a vector.deltaX and deltaY by the vector"s length (sqrt(deltaX*deltaX+deltaY*deltaY)), unless the length is 0.deltaX will now be the cosine of the angle between the vector and the horizontal axis (in the direction from the positive X to the positive Y axis at P1).deltaY will now be the sine of that angle.EDIT (Feb. 28, 2017): Even without normalizing (deltaX, deltaY):
deltaX will tell you whether the cosine described in step 3 is positive or negative.deltaY will tell you whether the sine described in step 4 is positive or negative.deltaX and deltaY will tell you which quadrant the angle is in, in relation to the positive X axis at P1:
+deltaX, +deltaY: 0 to 90 degrees.-deltaX, +deltaY: 90 to 180 degrees.-deltaX, -deltaY: 180 to 270 degrees (-180 to -90 degrees).+deltaX, -deltaY: 270 to 360 degrees (-90 to 0 degrees).An implementation in Python using radians (provided on July 19, 2015 by Eric Leschinski, who edited my answer):
from math import *
def angle_trunc(a):
while a < 0.0:
a += pi * 2
return a
def getAngleBetweenPoints(x_orig, y_orig, x_landmark, y_landmark):
deltaY = y_landmark - y_orig
deltaX = x_landmark - x_orig
return angle_trunc(atan2(deltaY, deltaX))
angle = getAngleBetweenPoints(5, 2, 1,4)
assert angle >= 0, "angle must be >= 0"
angle = getAngleBetweenPoints(1, 1, 2, 1)
assert angle == 0, "expecting angle to be 0"
angle = getAngleBetweenPoints(2, 1, 1, 1)
assert abs(pi - angle) <= 0.01, "expecting angle to be pi, it is: " + str(angle)
angle = getAngleBetweenPoints(2, 1, 2, 3)
assert abs(angle - pi/2) <= 0.01, "expecting angle to be pi/2, it is: " + str(angle)
angle = getAngleBetweenPoints(2, 1, 2, 0)
assert abs(angle - (pi+pi/2)) <= 0.01, "expecting angle to be pi+pi/2, it is: " + str(angle)
angle = getAngleBetweenPoints(1, 1, 2, 2)
assert abs(angle - (pi/4)) <= 0.01, "expecting angle to be pi/4, it is: " + str(angle)
angle = getAngleBetweenPoints(-1, -1, -2, -2)
assert abs(angle - (pi+pi/4)) <= 0.01, "expecting angle to be pi+pi/4, it is: " + str(angle)
angle = getAngleBetweenPoints(-1, -1, -1, 2)
assert abs(angle - (pi/2)) <= 0.01, "expecting angle to be pi/2, it is: " + str(angle)
All tests pass. See https://en.wikipedia.org/wiki/Unit_circle
Here"s a Python version:
from math import radians, cos, sin, asin, sqrt
def haversine(lon1, lat1, lon2, lat2):
"""
Calculate the great circle distance in kilometers between two points
on the earth (specified in decimal degrees)
"""
# convert decimal degrees to radians
lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])
# haversine formula
dlon = lon2 - lon1
dlat = lat2 - lat1
a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2
c = 2 * asin(sqrt(a))
r = 6371 # Radius of earth in kilometers. Use 3956 for miles. Determines return value units.
return c * r
I want to change a couple of files at one time, iff I can write to all of them. I"m wondering if I somehow can combine the multiple open calls with the with statement:
try:
with open("a", "w") as a and open("b", "w") as b:
do_something()
except IOError as e:
print "Operation failed: %s" % e.strerror
If that"s not possible, what would an elegant solution to this problem look like?
What is the best way to open a file as read/write if it exists, or if it does not, then create it and open it as read/write? From what I read, file = open("myfile.dat", "rw") should do this, right?
It is not working for me (Python 2.6.2) and I"m wondering if it is a version problem, or not supposed to work like that or what.
The bottom line is, I just need a solution for the problem. I am curious about the other stuff, but all I need is a nice way to do the opening part.
The enclosing directory was writeable by user and group, not other (I"m on a Linux system... so permissions 775 in other words), and the exact error was:
IOError: no such file or directory.
In the python built-in open function, what is the exact difference between the modes w, a, w+, a+, and r+?
In particular, the documentation implies that all of these will allow writing to the file, and says that it opens the files for "appending", "writing", and "updating" specifically, but does not define what these terms mean.
I am trying to implement a "Digit Recognition OCR" in OpenCV-Python (cv2). It is just for learning purposes. I would like to learn both KNearest and SVM features in OpenCV.
I have 100 samples (i.e. images) of each digit. I would like to train with them.
There is a sample letter_recog.py that comes with OpenCV sample. But I still couldn"t figure out on how to use it. I don"t understand what are the samples, responses etc. Also, it loads a txt file at first, which I didn"t understand first.
Later on searching a little bit, I could find a letter_recognition.data in cpp samples. I used it and made a code for cv2.KNearest in the model of letter_recog.py (just for testing):
import numpy as np
import cv2
fn = "letter-recognition.data"
a = np.loadtxt(fn, np.float32, delimiter=",", converters={ 0 : lambda ch : ord(ch)-ord("A") })
samples, responses = a[:,1:], a[:,0]
model = cv2.KNearest()
retval = model.train(samples,responses)
retval, results, neigh_resp, dists = model.find_nearest(samples, k = 10)
print results.ravel()
It gave me an array of size 20000, I don"t understand what it is.
Questions:
1) What is letter_recognition.data file? How to build that file from my own data set?
2) What does results.reval() denote?
3) How we can write a simple digit recognition tool using letter_recognition.data file (either KNearest or SVM)?
If you read an entire file with content = open("Path/to/file", "r").read() is the file handle left open until the script exits? Is there a more concise method to read a whole file?
I"m trying to make a system call in Python and store the output to a string that I can manipulate in the Python program.
#!/usr/bin/python
import subprocess
p2 = subprocess.Popen("ntpq -p")
I"ve tried a few things including some of the suggestions here:
Retrieving the output of subprocess.call()
but without any luck.
I am using Python 3.1 on a Windows 7 machine. Russian is the default system language, and utf-8 is the default encoding.
Looking at the answer to a previous question, I have attempting using the "codecs" module to give me a little luck. Here"s a few examples:
>>> g = codecs.open("C:UsersEricDesktopeeline.txt", "r", encoding="utf-8")
SyntaxError: (unicode error) "unicodeescape" codec can"t decode bytes in position 2-4: truncated UXXXXXXXX escape (<pyshell#39>, line 1)
>>> g = codecs.open("C:UsersEricDesktopSite.txt", "r", encoding="utf-8")
SyntaxError: (unicode error) "unicodeescape" codec can"t decode bytes in position 2-4: truncated UXXXXXXXX escape (<pyshell#40>, line 1)
>>> g = codecs.open("C:Python31Notes.txt", "r", encoding="utf-8")
SyntaxError: (unicode error) "unicodeescape" codec can"t decode bytes in position 11-12: malformed N character escape (<pyshell#41>, line 1)
>>> g = codecs.open("C:UsersEricDesktopSite.txt", "r", encoding="utf-8")
SyntaxError: (unicode error) "unicodeescape" codec can"t decode bytes in position 2-4: truncated UXXXXXXXX escape (<pyshell#44>, line 1)
My last idea was, I thought it might have been the fact that Windows "translates" a few folders, such as the "users" folder, into Russian (though typing "users" is still the correct path), so I tried it in the Python31 folder. Still, no luck. Any ideas?
I believe that running an external command with a slightly modified environment is a very common case. That"s how I tend to do it:
import subprocess, os
my_env = os.environ
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)
I"ve got a gut feeling that there"s a better way; does it look alright?
I have installed OpenCV on the Occidentalis operating system (a variant of Raspbian) on a Raspberry Pi, using jayrambhia"s script found here. It installed version 2.4.5.
When I try import cv2 in a Python program, I get the following message:
[email protected]~$ python cam.py
Traceback (most recent call last)
File "cam.py", line 1, in <module>
import cv2
ImportError: No module named cv2
The file cv2.so is stored in /usr/local/lib/python2.7/site-packages/...
There are also folders in /usr/local/lib called python3.2 and python2.6, which could be a problem but I"m not sure.
Is this a path error perhaps? Any help is appreciated, I am new to Linux.
How can I crop images, like I"ve done before in PIL, using OpenCV.
Working example on PIL
im = Image.open("0.png").convert("L")
im = im.crop((1, 1, 98, 33))
im.save("_0.png")
But how I can do it on OpenCV?
This is what I tried:
im = cv.imread("0.png", cv.CV_LOAD_IMAGE_GRAYSCALE)
(thresh, im_bw) = cv.threshold(im, 128, 255, cv.THRESH_OTSU)
im = cv.getRectSubPix(im_bw, (98, 33), (1, 1))
cv.imshow("Img", im)
cv.waitKey(0)
But it doesn"t work.
I think I incorrectly used getRectSubPix. If this is the case, please explain how I can correctly use this function.
Since this question was asked in 2010, there has been real simplification in how to do simple multithreading with Python with map and pool.
The code below comes from an article/blog post that you should definitely check out (no affiliation) - Parallelism in one line: A Better Model for Day to Day Threading Tasks. I"ll summarize below - it ends up being just a few lines of code:
from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(4)
results = pool.map(my_function, my_array)
Which is the multithreaded version of:
results = []
for item in my_array:
results.append(my_function(item))
Description
Map is a cool little function, and the key to easily injecting parallelism into your Python code. For those unfamiliar, map is something lifted from functional languages like Lisp. It is a function which maps another function over a sequence.
Map handles the iteration over the sequence for us, applies the function, and stores all of the results in a handy list at the end.
Implementation
Parallel versions of the map function are provided by two libraries:multiprocessing, and also its little known, but equally fantastic step child:multiprocessing.dummy.
multiprocessing.dummy is exactly the same as multiprocessing module, but uses threads instead (an important distinction - use multiple processes for CPU-intensive tasks; threads for (and during) I/O):
multiprocessing.dummy replicates the API of multiprocessing, but is no more than a wrapper around the threading module.
import urllib2
from multiprocessing.dummy import Pool as ThreadPool
urls = [
"http://www.python.org",
"http://www.python.org/about/",
"http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html",
"http://www.python.org/doc/",
"http://www.python.org/download/",
"http://www.python.org/getit/",
"http://www.python.org/community/",
"https://wiki.python.org/moin/",
]
# Make the Pool of workers
pool = ThreadPool(4)
# Open the URLs in their own threads
# and return the results
results = pool.map(urllib2.urlopen, urls)
# Close the pool and wait for the work to finish
pool.close()
pool.join()
And the timing results:
Single thread: 14.4 seconds
4 Pool: 3.1 seconds
8 Pool: 1.4 seconds
13 Pool: 1.3 seconds
Passing multiple arguments (works like this only in Python 3.3 and later):
To pass multiple arrays:
results = pool.starmap(function, zip(list_a, list_b))
Or to pass a constant and an array:
results = pool.starmap(function, zip(itertools.repeat(constant), list_a))
If you are using an earlier version of Python, you can pass multiple arguments via this workaround).
(Thanks to user136036 for the helpful comment.)
os.listdir()- list in the current directory
With listdir in os module you get the files and the folders in the current dir
import os
arr = os.listdir()
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
arr = os.listdir("c:\files")
globfrom glob
with glob you can specify a type of file to list like this
import glob
txtfiles = []
for file in glob.glob("*.txt"):
txtfiles.append(file)
glob in a list comprehensionmylist = [f for f in glob.glob("*.txt")]
import os
from os import listdir
from os.path import isfile, join
cwd = os.getcwd()
onlyfiles = [os.path.join(cwd, f) for f in os.listdir(cwd) if
os.path.isfile(os.path.join(cwd, f))]
print(onlyfiles)
["G:\getfilesname\getfilesname.py", "G:\getfilesname\example.txt"]
Getting the full path name with
os.path.abspath
You get the full path in return
import os
files_path = [os.path.abspath(x) for x in os.listdir()]
print(files_path)
["F:\documentiapplications.txt", "F:\documenticollections.txt"]
Walk: going through sub directories
os.walk returns the root, the directories list and the files list, that is why I unpacked them in r, d, f in the for loop; it, then, looks for other files and directories in the subfolders of the root and so on until there are no subfolders.
import os
# Getting the current work directory (cwd)
thisdir = os.getcwd()
# r=root, d=directories, f = files
for r, d, f in os.walk(thisdir):
for file in f:
if file.endswith(".docx"):
print(os.path.join(r, file))
os.listdir(): get files in the current directory (Python 2)
In Python 2, if you want the list of the files in the current directory, you have to give the argument as "." or os.getcwd() in the os.listdir method.
import os
arr = os.listdir(".")
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
To go up in the directory tree
# Method 1
x = os.listdir("..")
# Method 2
x= os.listdir("/")
Get files:
os.listdir()in a particular directory (Python 2 and 3)
import os
arr = os.listdir("F:\python")
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
Get files of a particular subdirectory with
os.listdir()
import os
x = os.listdir("./content")
os.walk(".")- current directory
import os
arr = next(os.walk("."))[2]
print(arr)
>>> ["5bs_Turismo1.pdf", "5bs_Turismo1.pptx", "esperienza.txt"]
next(os.walk("."))andos.path.join("dir", "file")
import os
arr = []
for d,r,f in next(os.walk("F:\_python")):
for file in f:
arr.append(os.path.join(r,file))
for f in arr:
print(files)
>>> F:\_python\dict_class.py
>>> F:\_python\programmi.txt
next(os.walk("F:\")- get the full path - list comprehension
[os.path.join(r,file) for r,d,f in next(os.walk("F:\_python")) for file in f]
>>> ["F:\_python\dict_class.py", "F:\_python\programmi.txt"]
os.walk- get full path - all files in sub dirs**
x = [os.path.join(r,file) for r,d,f in os.walk("F:\_python") for file in f]
print(x)
>>> ["F:\_python\dict.py", "F:\_python\progr.txt", "F:\_python\readl.py"]
os.listdir()- get only txt files
arr_txt = [x for x in os.listdir() if x.endswith(".txt")]
print(arr_txt)
>>> ["work.txt", "3ebooks.txt"]
Using
globto get the full path of the files
If I should need the absolute path of the files:
from path import path
from glob import glob
x = [path(f).abspath() for f in glob("F:\*.txt")]
for f in x:
print(f)
>>> F:acquistionline.txt
>>> F:acquisti_2018.txt
>>> F:ootstrap_jquery_ecc.txt
Using
os.path.isfileto avoid directories in the list
import os.path
listOfFiles = [f for f in os.listdir() if os.path.isfile(f)]
print(listOfFiles)
>>> ["a simple game.py", "data.txt", "decorator.py"]
Using
pathlibfrom Python 3.4
import pathlib
flist = []
for p in pathlib.Path(".").iterdir():
if p.is_file():
print(p)
flist.append(p)
>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> setup.py
>>> speak_gui2.py
>>> thumb.PNG
With list comprehension:
flist = [p for p in pathlib.Path(".").iterdir() if p.is_file()]
Alternatively, use pathlib.Path() instead of pathlib.Path(".")
Use glob method in pathlib.Path()
import pathlib
py = pathlib.Path().glob("*.py")
for file in py:
print(file)
>>> stack_overflow_list.py
>>> stack_overflow_list_tkinter.py
Get all and only files with os.walk
import os
x = [i[2] for i in os.walk(".")]
y=[]
for t in x:
for f in t:
y.append(f)
print(y)
>>> ["append_to_list.py", "data.txt", "data1.txt", "data2.txt", "data_180617", "os_walk.py", "READ2.py", "read_data.py", "somma_defaltdic.py", "substitute_words.py", "sum_data.py", "data.txt", "data1.txt", "data_180617"]
Get only files with next and walk in a directory
import os
x = next(os.walk("F://python"))[2]
print(x)
>>> ["calculator.bat","calculator.py"]
Get only directories with next and walk in a directory
import os
next(os.walk("F://python"))[1] # for the current dir use (".")
>>> ["python3","others"]
Get all the subdir names with
walk
for r,d,f in os.walk("F:\_python"):
for dirs in d:
print(dirs)
>>> .vscode
>>> pyexcel
>>> pyschool.py
>>> subtitles
>>> _metaprogramming
>>> .ipynb_checkpoints
os.scandir()from Python 3.5 and greater
import os
x = [f.name for f in os.scandir() if f.is_file()]
print(x)
>>> ["calculator.bat","calculator.py"]
# Another example with scandir (a little variation from docs.python.org)
# This one is more efficient than os.listdir.
# In this case, it shows the files only in the current directory
# where the script is executed.
import os
with os.scandir() as i:
for entry in i:
if entry.is_file():
print(entry.name)
>>> ebookmaker.py
>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> setup.py
>>> speakgui4.py
>>> speak_gui2.py
>>> speak_gui3.py
>>> thumb.PNG
Examples:
Ex. 1: How many files are there in the subdirectories?
In this example, we look for the number of files that are included in all the directory and its subdirectories.
import os
def count(dir, counter=0):
"returns number of files in dir and subdirs"
for pack in os.walk(dir):
for f in pack[2]:
counter += 1
return dir + " : " + str(counter) + "files"
print(count("F:\python"))
>>> "F:\python" : 12057 files"
Ex.2: How to copy all files from a directory to another?
A script to make order in your computer finding all files of a type (default: pptx) and copying them in a new folder.
import os
import shutil
from path import path
destination = "F:\file_copied"
# os.makedirs(destination)
def copyfile(dir, filetype="pptx", counter=0):
"Searches for pptx (or other - pptx is the default) files and copies them"
for pack in os.walk(dir):
for f in pack[2]:
if f.endswith(filetype):
fullpath = pack[0] + "\" + f
print(fullpath)
shutil.copy(fullpath, destination)
counter += 1
if counter > 0:
print("-" * 30)
print(" ==> Found in: `" + dir + "` : " + str(counter) + " files
")
for dir in os.listdir():
"searches for folders that starts with `_`"
if dir[0] == "_":
# copyfile(dir, filetype="pdf")
copyfile(dir, filetype="txt")
>>> _compiti18Compito Contabilità 1conti.txt
>>> _compiti18Compito Contabilità 1modula4.txt
>>> _compiti18Compito Contabilità 1moduloa4.txt
>>> ------------------------
>>> ==> Found in: `_compiti18` : 3 files
Ex. 3: How to get all the files in a txt file
In case you want to create a txt file with all the file names:
import os
mylist = ""
with open("filelist.txt", "w", encoding="utf-8") as file:
for eachfile in os.listdir():
mylist += eachfile + "
"
file.write(mylist)
Example: txt with all the files of an hard drive
"""
We are going to save a txt file with all the files in your directory.
We will use the function walk()
"""
import os
# see all the methods of os
# print(*dir(os), sep=", ")
listafile = []
percorso = []
with open("lista_file.txt", "w", encoding="utf-8") as testo:
for root, dirs, files in os.walk("D:\"):
for file in files:
listafile.append(file)
percorso.append(root + "\" + file)
testo.write(file + "
")
listafile.sort()
print("N. of files", len(listafile))
with open("lista_file_ordinata.txt", "w", encoding="utf-8") as testo_ordinato:
for file in listafile:
testo_ordinato.write(file + "
")
with open("percorso.txt", "w", encoding="utf-8") as file_percorso:
for file in percorso:
file_percorso.write(file + "
")
os.system("lista_file.txt")
os.system("lista_file_ordinata.txt")
os.system("percorso.txt")
All the file of C: in one text file
This is a shorter version of the previous code. Change the folder where to start finding the files if you need to start from another position. This code generate a 50 mb on text file on my computer with something less then 500.000 lines with files with the complete path.
import os
with open("file.txt", "w", encoding="utf-8") as filewrite:
for r, d, f in os.walk("C:\"):
for file in f:
filewrite.write(f"{r + file}
")
How to write a file with all paths in a folder of a type
With this function you can create a txt file that will have the name of a type of file that you look for (ex. pngfile.txt) with all the full path of all the files of that type. It can be useful sometimes, I think.
import os
def searchfiles(extension=".ttf", folder="H:\"):
"Create a txt file with all the file of a type"
with open(extension[1:] + "file.txt", "w", encoding="utf-8") as filewrite:
for r, d, f in os.walk(folder):
for file in f:
if file.endswith(extension):
filewrite.write(f"{r + file}
")
# looking for png file (fonts) in the hard disk H:
searchfiles(".png", "H:\")
>>> H:4bs_18Dolphins5.png
>>> H:4bs_18Dolphins6.png
>>> H:4bs_18Dolphins7.png
>>> H:5_18marketing htmlassetsimageslogo2.png
>>> H:7z001.png
>>> H:7z002.png
(New) Find all files and open them with tkinter GUI
I just wanted to add in this 2019 a little app to search for all files in a dir and be able to open them by doubleclicking on the name of the file in the list.
import tkinter as tk
import os
def searchfiles(extension=".txt", folder="H:\"):
"insert all files in the listbox"
for r, d, f in os.walk(folder):
for file in f:
if file.endswith(extension):
lb.insert(0, r + "\" + file)
def open_file():
os.startfile(lb.get(lb.curselection()[0]))
root = tk.Tk()
root.geometry("400x400")
bt = tk.Button(root, text="Search", command=lambda:searchfiles(".png", "H:\"))
bt.pack()
lb = tk.Listbox(root)
lb.pack(fill="both", expand=1)
lb.bind("<Double-Button>", lambda x: open_file())
root.mainloop()
I just used the following which was quite simple. First open a console then cd to where you"ve downloaded your file like some-package.whl and use
pip install some-package.whl
Note: if pip.exe is not recognized, you may find it in the "Scripts" directory from where python has been installed. If pip is not installed, this page can help: How do I install pip on Windows?
Note: for clarification
If you copy the *.whl file to your local drive (ex. C:some-dirsome-file.whl) use the following command line parameters --
pip install C:/some-dir/some-file.whl
This is the behaviour to adopt when the referenced object is deleted. It is not specific to Django; this is an SQL standard. Although Django has its own implementation on top of SQL. (1)
There are seven possible actions to take when such event occurs:
CASCADE: When the referenced object is deleted, also delete the objects that have references to it (when you remove a blog post for instance, you might want to delete comments as well). SQL equivalent: CASCADE.PROTECT: Forbid the deletion of the referenced object. To delete it you will have to delete all objects that reference it manually. SQL equivalent: RESTRICT.RESTRICT: (introduced in Django 3.1) Similar behavior as PROTECT that matches SQL"s RESTRICT more accurately. (See django documentation example)SET_NULL: Set the reference to NULL (requires the field to be nullable). For instance, when you delete a User, you might want to keep the comments he posted on blog posts, but say it was posted by an anonymous (or deleted) user. SQL equivalent: SET NULL.SET_DEFAULT: Set the default value. SQL equivalent: SET DEFAULT.SET(...): Set a given value. This one is not part of the SQL standard and is entirely handled by Django.DO_NOTHING: Probably a very bad idea since this would create integrity issues in your database (referencing an object that actually doesn"t exist). SQL equivalent: NO ACTION. (2)Source: Django documentation
See also the documentation of PostgreSQL for instance.
In most cases, CASCADE is the expected behaviour, but for every ForeignKey, you should always ask yourself what is the expected behaviour in this situation. PROTECT and SET_NULL are often useful. Setting CASCADE where it should not, can potentially delete all of your database in cascade, by simply deleting a single user.
Additional note to clarify cascade direction
It"s funny to notice that the direction of the CASCADE action is not clear to many people. Actually, it"s funny to notice that only the CASCADE action is not clear. I understand the cascade behavior might be confusing, however you must think that it is the same direction as any other action. Thus, if you feel that CASCADE direction is not clear to you, it actually means that on_delete behavior is not clear to you.
In your database, a foreign key is basically represented by an integer field which value is the primary key of the foreign object. Let"s say you have an entry comment_A, which has a foreign key to an entry article_B. If you delete the entry comment_A, everything is fine. article_B used to live without comment_A and don"t bother if it"s deleted. However, if you delete article_B, then comment_A panics! It never lived without article_B and needs it, and it"s part of its attributes (article=article_B, but what is article_B???). This is where on_delete steps in, to determine how to resolve this integrity error, either by saying:
PROTECT or RESTRICT in Django/SQL)SET_NULL)CASCADE behavior).SET_DEFAULT, or even SET(...)).DO_NOTHING)I hope it makes cascade direction clearer. :)
Footnotes
(1) Django has its own implementation on top of SQL. And, as mentioned by @JoeMjr2 in the comments below, Django will not create the SQL constraints. If you want the constraints to be ensured by your database (for instance, if your database is used by another application, or if you hang in the database console from time to time), you might want to set the related constraints manually yourself. There is an open ticket to add support for database-level on delete constrains in Django.
(2) Actually, there is one case where
DO_NOTHINGcan be useful: If you want to skip Django"s implementation and implement the constraint yourself at the database-level.
Running brew reinstall [email protected] didn"t work for my existing Python 2.7 virtual environments. Inside them there were still ERROR:root:code for hash sha1 was not found errors.
I encountered this problem after I ran brew upgrade openssl. And here"s the fix:
$ ls /usr/local/Cellar/openssl
...which shows
1.0.2t
According to the existing version, run:
$ brew switch openssl 1.0.2t
...which shows
Cleaning /usr/local/Cellar/openssl/1.0.2t
Opt link created for /usr/local/Cellar/openssl/1.0.2t
After that, run the following command in a Python 2.7 virtualenv:
(my-venv) $ python -c "import hashlib;m=hashlib.md5();print(m.hexdigest())"
...which shows
d41d8cd98f00b204e9800998ecf8427e
No more errors.
You opened the file in binary mode:
with open(fname, "rb") as f:
This means that all data read from the file is returned as bytes objects, not str. You cannot then use a string in a containment test:
if "some-pattern" in tmp: continue
You"d have to use a bytes object to test against tmp instead:
if b"some-pattern" in tmp: continue
or open the file as a textfile instead by replacing the "rb" mode with "r".
⚡️ TL;DR — One line solution.
All you have to do is:
sudo easy_install pip
2019: ⚠️
easy_installhas been deprecated. Check Method #2 below for preferred installation!
Details:
⚡️ OK, I read the solutions given above, but here"s an EASY solution to install
pip.
MacOS comes with Python installed. But to make sure that you have Python installed open the terminal and run the following command.
python --version
If this command returns a version number that means Python exists. Which also means that you already have access to easy_install considering you are using macOS/OSX.
ℹ️ Now, all you have to do is run the following command.
sudo easy_install pip
After that, pip will be installed and you"ll be able to use it for installing other packages.
Let me know if you have any problems installing pip this way.
Cheers!
P.S. I ended up blogging a post about it. QuickTip: How Do I Install pip on macOS or OS X?
✅ UPDATE (Jan 2019): METHOD #2: Two line solution —
easy_install has been deprecated. Please use get-pip.py instead.
First of all download the get-pip file
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
Now run this file to install pip
python get-pip.py
That should do it.
Another gif you said? Here ya go!
I noticed that every now and then I need to Google fopen all over again, just to build a mental image of what the primary differences between the modes are. So, I thought a diagram will be faster to read next time. Maybe someone else will find that helpful too.
It helps to install a python package foo on your machine (can also be in virtualenv) so that you can import the package foo from other projects and also from [I]Python prompts.
It does the similar job of pip, easy_install etc.,
Using setup.py
Let"s start with some definitions:
Package - A folder/directory that contains __init__.py file.
Module - A valid python file with .py extension.
Distribution - How one package relates to other packages and modules.
Let"s say you want to install a package named foo. Then you do,
$ git clone https://github.com/user/foo
$ cd foo
$ python setup.py install
Instead, if you don"t want to actually install it but still would like to use it. Then do,
$ python setup.py develop
This command will create symlinks to the source directory within site-packages instead of copying things. Because of this, it is quite fast (particularly for large packages).
Creating setup.py
If you have your package tree like,
foo
├── foo
│   ├── data_struct.py
│   ├── __init__.py
│   └── internals.py
├── README
├── requirements.txt
└── setup.py
Then, you do the following in your setup.py script so that it can be installed on some machine:
from setuptools import setup
setup(
name="foo",
version="1.0",
description="A useful module",
author="Man Foo",
author_email="[email protected]",
packages=["foo"], #same as name
install_requires=["wheel", "bar", "greek"], #external packages as dependencies
)
Instead, if your package tree is more complex like the one below:
foo
├── foo
│   ├── data_struct.py
│   ├── __init__.py
│   └── internals.py
├── README
├── requirements.txt
├── scripts
│   ├── cool
│   └── skype
└── setup.py
Then, your setup.py in this case would be like:
from setuptools import setup
setup(
name="foo",
version="1.0",
description="A useful module",
author="Man Foo",
author_email="[email protected]",
packages=["foo"], #same as name
install_requires=["wheel", "bar", "greek"], #external packages as dependencies
scripts=[
"scripts/cool",
"scripts/skype",
]
)
Add more stuff to (setup.py) & make it decent:
from setuptools import setup
with open("README", "r") as f:
long_description = f.read()
setup(
name="foo",
version="1.0",
description="A useful module",
license="MIT",
long_description=long_description,
author="Man Foo",
author_email="[email protected]",
url="http://www.foopackage.com/",
packages=["foo"], #same as name
install_requires=["wheel", "bar", "greek"], #external packages as dependencies
scripts=[
"scripts/cool",
"scripts/skype",
]
)
The long_description is used in pypi.org as the README description of your package.
And finally, you"re now ready to upload your package to PyPi.org so that others can install your package using pip install yourpackage.
At this point there are two options.
Once your package name is registered in pypi.org, nobody can claim or use it. Python packaging suggests the twine package for uploading purposes (of your package to PyPi). Thus,
(1) the first step is to locally build the distributions using:
# prereq: wheel (pip install wheel)
$ python setup.py sdist bdist_wheel
(2) then using twine for uploading either to test.pypi.org or pypi.org:
$ twine upload --repository testpypi dist/*
username: ***
password: ***
It will take few minutes for the package to appear on test.pypi.org. Once you"re satisfied with it, you can then upload your package to the real & permanent index of pypi.org simply with:
$ twine upload dist/*
Optionally, you can also sign the files in your package with a GPG by:
$ twine upload dist/* --sign
Bonus Reading:
See a sample setup.py from a real project here: torchvision-setup.py
reload hacksWithout seeing the source it"s difficult to know the root cause, so I"ll have to speak generally.
UnicodeDecodeError: "ascii" codec can"t decode byte generally happens when you try to convert a Python 2.x str that contains non-ASCII to a Unicode string without specifying the encoding of the original string.
In brief, Unicode strings are an entirely separate type of Python string that does not contain any encoding. They only hold Unicode point codes and therefore can hold any Unicode point from across the entire spectrum. Strings contain encoded text, beit UTF-8, UTF-16, ISO-8895-1, GBK, Big5 etc. Strings are decoded to Unicode and Unicodes are encoded to strings. Files and text data are always transferred in encoded strings.
The Markdown module authors probably use unicode() (where the exception is thrown) as a quality gate to the rest of the code - it will convert ASCII or re-wrap existing Unicodes strings to a new Unicode string. The Markdown authors can"t know the encoding of the incoming string so will rely on you to decode strings to Unicode strings before passing to Markdown.
Unicode strings can be declared in your code using the u prefix to strings. E.g.
>>> my_u = u"my ünicôdé strįng"
>>> type(my_u)
<type "unicode">
Unicode strings may also come from file, databases and network modules. When this happens, you don"t need to worry about the encoding.
Conversion from str to Unicode can happen even when you don"t explicitly call unicode().
The following scenarios cause UnicodeDecodeError exceptions:
# Explicit conversion without encoding
unicode("€")
# New style format string into Unicode string
# Python will try to convert value string to Unicode first
u"The currency is: {}".format("€")
# Old style format string into Unicode string
# Python will try to convert value string to Unicode first
u"The currency is: %s" % "€"
# Append string to Unicode
# Python will try to convert string to Unicode first
u"The currency is: " + "€"
In the following diagram, you can see how the word café has been encoded in either "UTF-8" or "Cp1252" encoding depending on the terminal type. In both examples, caf is just regular ascii. In UTF-8, é is encoded using two bytes. In "Cp1252", é is 0xE9 (which is also happens to be the Unicode point value (it"s no coincidence)). The correct decode() is invoked and conversion to a Python Unicode is successfull:
In this diagram, decode() is called with ascii (which is the same as calling unicode() without an encoding given). As ASCII can"t contain bytes greater than 0x7F, this will throw a UnicodeDecodeError exception:
It"s good practice to form a Unicode sandwich in your code, where you decode all incoming data to Unicode strings, work with Unicodes, then encode to strs on the way out. This saves you from worrying about the encoding of strings in the middle of your code.
If you need to bake non-ASCII into your source code, just create Unicode strings by prefixing the string with a u. E.g.
u"Zürich"
To allow Python to decode your source code, you will need to add an encoding header to match the actual encoding of your file. For example, if your file was encoded as "UTF-8", you would use:
# encoding: utf-8
This is only necessary when you have non-ASCII in your source code.
Usually non-ASCII data is received from a file. The io module provides a TextWrapper that decodes your file on the fly, using a given encoding. You must use the correct encoding for the file - it can"t be easily guessed. For example, for a UTF-8 file:
import io
with io.open("my_utf8_file.txt", "r", encoding="utf-8") as my_file:
my_unicode_string = my_file.read()
my_unicode_string would then be suitable for passing to Markdown. If a UnicodeDecodeError from the read() line, then you"ve probably used the wrong encoding value.
The Python 2.7 CSV module does not support non-ASCII characters üò©. Help is at hand, however, with https://pypi.python.org/pypi/backports.csv.
Use it like above but pass the opened file to it:
from backports import csv
import io
with io.open("my_utf8_file.txt", "r", encoding="utf-8") as my_file:
for row in csv.reader(my_file):
yield row
Most Python database drivers can return data in Unicode, but usually require a little configuration. Always use Unicode strings for SQL queries.
MySQLIn the connection string add:
charset="utf8",
use_unicode=True
E.g.
>>> db = MySQLdb.connect(host="localhost", user="root", passwd="passwd", db="sandbox", use_unicode=True, charset="utf8")
Add:
psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)
Web pages can be encoded in just about any encoding. The Content-type header should contain a charset field to hint at the encoding. The content can then be decoded manually against this value. Alternatively, Python-Requests returns Unicodes in response.text.
If you must decode strings manually, you can simply do my_string.decode(encoding), where encoding is the appropriate encoding. Python 2.x supported codecs are given here: Standard Encodings. Again, if you get UnicodeDecodeError then you"ve probably got the wrong encoding.
Work with Unicodes as you would normal strs.
print writes through the stdout stream. Python tries to configure an encoder on stdout so that Unicodes are encoded to the console"s encoding. For example, if a Linux shell"s locale is en_GB.UTF-8, the output will be encoded to UTF-8. On Windows, you will be limited to an 8bit code page.
An incorrectly configured console, such as corrupt locale, can lead to unexpected print errors. PYTHONIOENCODING environment variable can force the encoding for stdout.
Just like input, io.open can be used to transparently convert Unicodes to encoded byte strings.
The same configuration for reading will allow Unicodes to be written directly.
Python 3 is no more Unicode capable than Python 2.x is, however it is slightly less confused on the topic. E.g the regular str is now a Unicode string and the old str is now bytes.
The default encoding is UTF-8, so if you .decode() a byte string without giving an encoding, Python 3 uses UTF-8 encoding. This probably fixes 50% of people"s Unicode problems.
Further, open() operates in text mode by default, so returns decoded str (Unicode ones). The encoding is derived from your locale, which tends to be UTF-8 on Un*x systems or an 8-bit code page, such as windows-1251, on Windows boxes.
sys.setdefaultencoding("utf8")It"s a nasty hack (there"s a reason you have to use reload) that will only mask problems and hinder your migration to Python 3.x. Understand the problem, fix the root cause and enjoy Unicode zen.
See Why should we NOT use sys.setdefaultencoding("utf-8") in a py script? for further details