👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!
I"ve been testing out Selenium with Chromedriver and I noticed that some pages can detect that you"re using Selenium even though there"s no automation at all. Even when I"m just browsing manually just using Chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I"ve checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal Chrome browser.
When I browse to these sites in normal Chrome everything works fine, but the moment I use Selenium I"m detected.
In theory, chromedriver and Chrome should look literally exactly the same to any webserver, but somehow they can detect it.
If you want some testcode try out this:
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--profile-directory=Default")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print "arguments done"
driver.get("http://stubhub.com")
If you browse around stubhub you"ll get redirected and "blocked" within one or two requests. I"ve been investigating this and I can"t figure out how they can tell that a user is using Selenium.
How do they do it?
I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal Firefox browser with only the additional plugin.
When I use Fiddler to view the HTTP requests being sent back and forth I"ve noticed that the "fake browser"s" requests often have "no-cache" in the response header.
Results like this Is there a way to detect that I'm in a Selenium Webdriver page from JavaScript suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.
The site uploads a fingerprint to their servers, but I checked and the fingerprint of Selenium is identical to the fingerprint when using Chrome.
This is one of the fingerprint payloads that they send to their servers: {"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionModule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContentDecryptionModuleapplication/x-ppapi-widevine-cdm","4":"NativeClientExecutableapplication/x-nacl","5":"PortableNativeClientExecutableapplication/x-pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"TrebuchetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationMono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}
It"s identical in Selenium and in Chrome.
VPNs work for a single use, but they get detected after I load the first page. Clearly some JavaScript is being run to detect Selenium.
👻 Read also: what is the best laptop for engineering students?
Can a website detect when you are using Selenium with chromedriver? __del__: Questions
How can I make a time delay in Python?
5 answers
I would like to know how to put a time delay in a Python script.
Answer #1
import time
time.sleep(5) # Delays for 5 seconds. You can also use a float value.
Here is another example where something is run approximately once a minute:
import time
while True:
print("This prints once a minute.")
time.sleep(60) # Delay for 1 minute (60 seconds).
Answer #2
You can use the sleep()
function in the time
module. It can take a float argument for sub-second resolution.
from time import sleep
sleep(0.1) # Time in seconds
Can a website detect when you are using Selenium with chromedriver? __del__: Questions
How to delete a file or folder in Python?
5 answers
How do I delete a file or folder in Python?
Answer #1
os.remove()
removes a file.os.rmdir()
removes an empty directory.shutil.rmtree()
deletes a directory and all its contents.
Path
objects from the Python 3.4+ pathlib
module also expose these instance methods:
pathlib.Path.unlink()
removes a file or symbolic link.pathlib.Path.rmdir()
removes an empty directory.
Answer #2
os.remove()
removes a file.os.rmdir()
removes an empty directory.shutil.rmtree()
deletes a directory and all its contents.
Path
objects from the Python 3.4+ pathlib
module also expose these instance methods:
pathlib.Path.unlink()
removes a file or symbolic link.pathlib.Path.rmdir()
removes an empty directory.
Answer #3
Python syntax to delete a file
import os
os.remove("/tmp/<file_name>.txt")
Or
import os
os.unlink("/tmp/<file_name>.txt")
Or
pathlib Library for Python version >= 3.4
file_to_rem = pathlib.Path("/tmp/<file_name>.txt")
file_to_rem.unlink()
Path.unlink(missing_ok=False)
Unlink method used to remove the file or the symbolik link.
If missing_ok is false (the default), FileNotFoundError is raised if the path does not exist.
If missing_ok is true, FileNotFoundError exceptions will be ignored (same behavior as the POSIX rm -f command).
Changed in version 3.8: The missing_ok parameter was added.
Best practice
- First, check whether the file or folder exists or not then only delete that file. This can be achieved in two ways :
a.os.path.isfile("/path/to/file")
b. Useexception handling.
EXAMPLE for os.path.isfile
#!/usr/bin/python
import os
myfile="/tmp/foo.txt"
## If file exists, delete it ##
if os.path.isfile(myfile):
os.remove(myfile)
else: ## Show an error ##
print("Error: %s file not found" % myfile)
Exception Handling
#!/usr/bin/python
import os
## Get input ##
myfile= raw_input("Enter file name to delete: ")
## Try to delete the file ##
try:
os.remove(myfile)
except OSError as e: ## if failed, report it back to the user ##
print ("Error: %s - %s." % (e.filename, e.strerror))
RESPECTIVE OUTPUT
Enter file name to delete : demo.txt Error: demo.txt - No such file or directory. Enter file name to delete : rrr.txt Error: rrr.txt - Operation not permitted. Enter file name to delete : foo.txt
Python syntax to delete a folder
shutil.rmtree()
Example for shutil.rmtree()
#!/usr/bin/python
import os
import sys
import shutil
# Get directory name
mydir= raw_input("Enter directory name: ")
## Try to remove tree; if failed show an error using try...except on screen
try:
shutil.rmtree(mydir)
except OSError as e:
print ("Error: %s - %s." % (e.filename, e.strerror))
Is there a simple way to delete a list element by value?
5 answers
I want to remove a value from a list if it exists in the list (which it may not).
a = [1, 2, 3, 4]
b = a.index(6)
del a[b]
print(a)
The above case (in which it does not exist) shows the following error:
Traceback (most recent call last):
File "D:zjm_codea.py", line 6, in <module>
b = a.index(6)
ValueError: list.index(x): x not in list
So I have to do this:
a = [1, 2, 3, 4]
try:
b = a.index(6)
del a[b]
except:
pass
print(a)
But is there not a simpler way to do this?
Answer #1
To remove an element"s first occurrence in a list, simply use list.remove
:
>>> a = ["a", "b", "c", "d"]
>>> a.remove("b")
>>> print(a)
["a", "c", "d"]
Mind that it does not remove all occurrences of your element. Use a list comprehension for that.
>>> a = [10, 20, 30, 40, 20, 30, 40, 20, 70, 20]
>>> a = [x for x in a if x != 20]
>>> print(a)
[10, 30, 40, 30, 40, 70]
We hope this article has helped you to resolve the problem. Apart from Can a website detect when you are using Selenium with chromedriver?, check other __del__-related topics.
Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.
By the way, this material is also available in other languages:
- Italiano Can a website detect when you are using Selenium with chromedriver?
- Deutsch Can a website detect when you are using Selenium with chromedriver?
- Français Can a website detect when you are using Selenium with chromedriver?
- Español Can a website detect when you are using Selenium with chromedriver?
- Türk Can a website detect when you are using Selenium with chromedriver?
- Русский Can a website detect when you are using Selenium with chromedriver?
- Português Can a website detect when you are using Selenium with chromedriver?
- Polski Can a website detect when you are using Selenium with chromedriver?
- Nederlandse Can a website detect when you are using Selenium with chromedriver?
- 中文 Can a website detect when you are using Selenium with chromedriver?
- 한국어 Can a website detect when you are using Selenium with chromedriver?
- 日本語 Can a website detect when you are using Selenium with chromedriver?
- हिन्दी Can a website detect when you are using Selenium with chromedriver?
San Francisco | 2023-03-29
Simply put and clear. Thank you for sharing. Can a website detect when you are using Selenium with chromedriver? and other issues with Python functions was always my weak point 😁. Will use it in my bachelor thesis
California | 2023-03-29
Simply put and clear. Thank you for sharing. Can a website detect when you are using Selenium with chromedriver? and other issues with dis Python module was always my weak point 😁. I am just not quite sure it is the best method
Milan | 2023-03-29
Simply put and clear. Thank you for sharing. Can a website detect when you are using Selenium with chromedriver? and other issues with COM PHP module was always my weak point 😁. I just hope that will not emerge anymore