Backend De Javascript: StackOverflow Questions

JSON datetime between Python and JavaScript

Question by kevin

I want to send a datetime.datetime object in serialized form from Python using JSON and de-serialize in JavaScript using JSON. What is the best way to do this?

Javascript equivalent of Python"s zip function

Is there a javascript equivalent of Python"s zip function? That is, given multiple arrays of equal lengths create an array of pairs.

For instance, if I have three arrays that look like this:

var array1 = [1, 2, 3];
var array2 = ["a","b","c"];
var array3 = [4, 5, 6];

The output array should be:

var output array:[[1,"a",4], [2,"b",5], [3,"c",6]]

What blocks Ruby, Python to get Javascript V8 speed?

Are there any Ruby / Python features that are blocking implementation of optimizations (e.g. inline caching) V8 engine has?

Python is co-developed by Google guys so it shouldn"t be blocked by software patents.

Or this is rather matter of resources put into the V8 project by Google.

Django Template Variables and Javascript

When I render a page using the Django template renderer, I can pass in a dictionary variable containing various values to manipulate them in the page using {{ myVar }}.

Is there a way to access the same variable in Javascript (perhaps using the DOM, I don"t know how Django makes the variables accessible)? I want to be able to lookup details using an AJAX lookup based on the values contained in the variables passed in.

Web-scraping JavaScript page with Python

I"m trying to develop a simple web scraper. I want to extract text without the HTML code. In fact, I achieve this goal, but I have seen that in some pages where JavaScript is loaded I didn"t obtain good results.

For example, if some JavaScript code adds some text, I can"t see it, because when I call

response = urllib2.urlopen(request)

I get the original text without the added one (because JavaScript is executed in the client).

So, I"m looking for some ideas to solve this problem.

How can I pass data from Flask to JavaScript in a template?

My app makes a call to an API that returns a dictionary. I want to pass information from this dict to JavaScript in the view. I am using the Google Maps API in the JS, specifically, so I"d like to pass it a list of tuples with the long/lat information. I know that render_template will pass these variables to the view so they can be used in HTML, but how could I pass them to JavaScript in the template?

from flask import Flask
from flask import render_template

app = Flask(__name__)

import foo_api

api = foo_api.API("API KEY")

@app.route("/")
def get_data():
    events = api.call(get_event, arg0, arg1)
    geocode = event["latitude"], event["longitude"]
    return render_template("get_data.html", geocode=geocode)

WebDriver click() vs JavaScript click()

The Story:

Here on StackOverflow, I"ve seen users reporting that they cannot click an element via selenium WebDriver "click" command and can work around it with a JavaScript click by executing a script.

Example in Python:

element = driver.find_element_by_id("myid")
driver.execute_script("arguments[0].click();", element)

Example in WebDriverJS/Protractor:

var elm = $("#myid");
browser.executeScript("arguments[0].click();", elm.getWebElement());

The Question:

Why is clicking "via JavaScript" works when a regular WebDriver click does not? When exactly is this happening and what is the downside of this workaround (if any)?

^{I personally used this workaround without fully understanding why I have to do it and what problems it can lead to.}

are there dictionaries in javascript like python?

i need to make a dictionary in javascript like this

i dont remember the exact notation, but it was something like:

states_dictionary={ CT=[alex,harry], AK=[liza,alex], TX=[fred, harry] ........ }

is there such a thing in javascript?

Python Pandas equivalent in JavaScript

With this CSV example:

   Source,col1,col2,col3
   foo,1,2,3
   bar,3,4,5

The standard method I use Pandas is this:

1. Parse CSV

2. Select columns into a data frame (col1 and col3)

3. Process the column (e.g. avarage the values of col1 and col3)

Is there a JavaScript library that does that like Pandas?

Is there a JavaScript equivalent of the Python pass statement that does nothing?

I am looking for a JavaScript equivalent of the Python:

pass statement that does not run the function of the ... notation?

Is there such a thing in JavaScript?

Answer #1

from IPython.display import HTML

HTML("""<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $("div.input").hide();
 } else {
 $("div.input").show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>""")

Answer #2

Basically, the way the Selenium detection works, is that they test for predefined JavaScript variables which appear when running with Selenium. The bot detection scripts usually look anything containing word "selenium" / "webdriver" in any of the variables (on window object), and also document variables called $cdc_ and $wdc_. Of course, all of this depends on which browser you are on. All the different browsers expose different things.

For me, I used Chrome, so, all that I had to do was to ensure that $cdc_ didn"t exist anymore as a document variable, and voil√† (download chromedriver source code, modify chromedriver and re-compile $cdc_ under different name.)

This is the function I modified in chromedriver:

File call_function.js:

function getPageCache(opt_doc) {
  var doc = opt_doc || document;
  //var key = "$cdc_asdjflasutopfhvcZLmcfl_";
  var key = "randomblabla_";
  if (!(key in doc))
    doc[key] = new Cache();
  return doc[key];
}

(Note the comment. All I did I turned $cdc_ to randomblabla_.)

Here is pseudocode which demonstrates some of the techniques that bot networks might use:

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window["document"][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window["document"]) {
        if (documentKey.match(/$[a-z]dc_/) && window["document"][documentKey]["cache_"]) {
            return true;
        }
    }

    if (window["external"] && window["external"].toString() && (window["external"].toString()["indexOf"]("Sequentum") != -1)) return true;

    if (window["document"]["documentElement"]["getAttribute"]("selenium")) return true;
    if (window["document"]["documentElement"]["getAttribute"]("webdriver")) return true;
    if (window["document"]["documentElement"]["getAttribute"]("driver")) return true;

    return false;
};

According to user szx, it is also possible to simply open chromedriver.exe in a hex editor, and just do the replacement manually, without actually doing any compiling.

Answer #3

Python"s pass mainly exists because in Python whitespace matters within a block. In Javascript, the equivalent would be putting nothing within the block, i.e. {}.

Answer #4

Edit 2: As of March 2021, none of these methods will work as google added a captcha button that randomly pops up after some time.

Edit 1: Apparently the solution is very easy, and doesn"t need any JavaScript. Just create a new cell at the bottom having the following line:

while True:pass

now keep the cell in the run sequence so that the infinite loop won"t stop and thus keep your session alive.

Old method: Set a javascript interval to click on the connect button every 60 seconds. Open developer-settings (in your web-browser) with Ctrl+Shift+I then click on console tab and type this on the console prompt. (for mac press Option+Command+I)

function ConnectButton(){
    console.log("Connect pushed"); 
    document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click() 
}
setInterval(ConnectButton,60000);

Answer #5

Contrarily to what the currently accepted answer suggests, there"s nothing specific to PhantomJS when it comes to the difference between having WebDriver do a click and doing it in JavaScript.

The Difference

The essential difference between the two methods is common to all browsers and can be explained pretty simply:

• WebDriver: When WebDriver does the click, it attempts as best as it can to simulate what happens when a real user uses the browser. Suppose you have an element A which is a button that says "Click me" and an element B which is a div element which is transparent but has its dimensions and zIndex set so that it completely covers A. Then you tell WebDriver to click A. WebDriver will simulate the click so that B receives the click first. Why? Because B covers A, and if a user were to try to click on A, then B would get the event first. Whether or not A would eventually get the click event depends on how B handles the event. At any rate, the behavior with WebDriver in this case is the same as when a real user tries to click on A.

• JavaScript: Now, suppose you use JavaScript to do A.click(). This method of clicking does not reproduce what really happens when the user tries to click A. JavaScript sends the click event directly to A, and B will not get any event.

Why a JavaScript Click Works When a WebDriver Click Does Not?

As I mentioned above WebDriver will try to simulate as best it can what happens when a real user is using a browser. The fact of the matter is that the DOM can contain elements that a user cannot interact with, and WebDriver won"t allow you to click on these element. Besides the overlapping case I mentioned, this also entails that invisible elements cannot be clicked. A common case I see in Stack Overflow questions is someone who is trying to interact with a GUI element that already exists in the DOM but becomes visible only when some other element has been manipulated. This sometimes happens with dropdown menus: you have to first click on the button the brings up the dropdown before a menu item can be selected. If someone tries to click the menu item before the menu is visible, WebDriver will balk and say that the element cannot be manipulated. If the person then tries to do it with JavaScript, it will work because the event is delivered directly to the element, irrespective of visibility.

When Should You Use JavaScript for Clicking?

If you are using Selenium for testing an application, my answer to this question is "almost never". By and large, your Selenium test should reproduce what a user would do with the browser. Taking the example of the drop down menu: a test should click on the button that brings up the drop down first, and then click on the menu item. If there is a problem with the GUI because the button is invisible, or the button fails to show the menu items, or something similar, then your test will fail and you"ll have detected the bug. If you use JavaScript to click around, you won"t be able to detect these bugs through automated testing.

I say "almost never" because there may be exceptions where it makes sense to use JavaScript. They should be very rare, though.

If you are using Selenium for scraping sites, then it is not as critical to attempt to reproduce user behavior. So using JavaScript to bypass the GUI is less of an issue.

Answer #6

WebAssembly vs asm.js

First, let"s take a look how, in principle, WebAssembly is different from asm.js, and whether there"s potential to reuse existing knowledge and tooling. The following gives pretty good overview:

• Why create a new standard when there is already asm.js?
• What is the difference between asm.js and web assembly?
• Why WebAssembly is Faster Than asm.js

Let"s recapitulate, WebAssembly (MVP, as there"s more on its roadmap, roughly):

• is a binary format of AST with static typing, which can be executed by existing JavaScript engines (and thus JIT-able or compiled AOT),
• it"s 10-20% more compact (gzipped comparison) and an order of magnitude faster to parse than JavaScript,
• it can express more low-level operation that won"t fit into JavaScript syntax, read asm.js (e.g. 64-bit integers, special CPU instructions, SIMD, etc)
• is convertible (to some extent) to/from asm.js.

Thus, currently WebAssembly is an iteration on asm.js and targets only C/C++ (and similar languages).

Python on the Web

It doesn"t look like GC is the only thing that stops Python code from targeting WebAssembly/asm.js. Both represent low-level statically typed code, in which Python code can"t (realistically) be represented. As current toolchain of WebAssembly/asm.js is based on LLVM, a language that can be easily compiled to LLVM IR can be converted to WebAssembly/asm.js. But alas, Python is too dynamic to fit into it as well, as proven by Unladen Swallow and several attempts of PyPy.

This asm.js presentation has slides about the state of dynamic languages. What it means is that currently it"s only possible to compile whole VM (language implementation in C/C++) to WebAssembly/asm.js and interpret (with JIT where possible) original sources. For Python there"re several existing projects:

1. PyPy: PyPy.js (author"s talk at PyCon). Here"s release repo. Main JS file, pypyjs.vm.js, is 13 MB (2MB after gzip -6) + Python stdlib + other stuff.

2. CPython: pyodide, EmPython, CPython-Emscripten, EmCPython, etc. empython.js is 5.8 MB (2.1 MB after gzip -6), no stdlib.

3. Micropython: this fork.

   There was no built JS file there, so I was able to build it with trzeci/emscripten/, a ready-made Emscripten toolchain. Something like:

    git clone https://github.com/matthewelse/micropython.git
    cd micropython
    docker run --rm -it -v $(pwd):/src trzeci/emscripten bash
    apt-get update && apt-get install -y python3
    cd emscripten
    make -j
    # to run REPL: npm install && nodejs server.js 
   

   It produces micropython.js of 1.1 MB (225 KB after gzip -d). The latter is already something to consider, if you need only very compliant implementation without stdlib.

   To produce WebAssembly build you can change line 13 of the Makefile to

    CC = emcc -s RESERVED_FUNCTION_POINTERS=20 -s WASM=1
   

   Then make -j produces:

    113 KB micropython.js
    240 KB micropython.wasm
   

   You can look at HTML output of emcc hello.c -s WASM=1 -o hello.html, to see how to use these files.

   This way you can also potentially build PyPy and CPython in WebAssembly to interpret your Python application in a compliant browser.

Another potentially interesting thing here is Nuitka, a Python to C++ compiler. Potentially it can be possible to build your Python app to C++ and then compile it along with CPython with Emscripten. But practically I"ve no idea how to do it.

Solutions

For the time being, if you"re building a conventional web site or web app where download several-megabyte JS file is barely an option, take a look at Python-to-JavaScript transpilers (e.g. Transcrypt) or JavaScript Python implementations (e.g. Brython). Or try your luck with others from list of languages that compile to JavaScript.

Otherwise, if download size is not an issue, and you"re ready to tackle a lot of rough edges, choose between the three above.

Q3 2020 update

1. JavaScript port was integrated into MicroPython. It lives in ports/javascript.

2. The port is available as a npm package called MicroPython.js. You can try it out in RunKit.

3. There"s an actively developed Python implementation in Rust, called RustPython. Because Rust officially supports WebAssembly as compile target, no surprise there"s demo link right in the top of the readme. Though, it"s early. Their disclaimer follows.

   RustPython is in a development phase and should not be used in production or a fault intolerant setting.

   Our current build supports only a subset of Python syntax.

Answer #7

This wiki will summarize and compare many pandas-like Javascript libraries.

In general, you should check out the d3 Javascript library. d3 is very useful "swiss army knife" for handling data in Javascript, just like pandas is helpful for Python. You may see d3 used frequently like pandas, even if d3 is not exactly a DataFrame/Pandas replacement (i.e. d3 doesn"t have the same API; d3 doesn"t have Series / DataFrame which behave like in pandas)

Ahmed"s answer explains how d3 can be used to achieve some DataFrame functionality, and some of the libraries below were inspired by things like LearnJsData which uses d3 and lodash.

As for DataFrame-style data transformation (splitting, joining, group by etc) , here is a quick list of some of the Javascript libraries.

Note some libraries are Node.js aka Server-side Javascript, some are browser-compatible aka client-side Javascript, and some are Typescript. So use the option that"s right for you.

• danfo-js
  • From Vignesh"s answer
  • danfo (which is often imported and aliased as dfd); has a basic DataFrame-type data structure, with the ability to plot directly
  • Built by the team at Tensorflow: "One of the main goals of Danfo.js is to bring data processing, machine learning and AI tools to JavaScript developers. ... Open-source libraries like Numpy and Pandas..."
  • pandas is built on top of numpy; likewise danfo-js is built on tensorflow-js
• pandas-js
  • UPDATE The pandas-js repo has not been updated in awhile
  • From STEEL and Feras" answers
  • "pandas.js is an open source (experimental) library mimicking the Python pandas library. It relies on Immutable.js as the NumPy logical equivalent. The main data objects in pandas.js are, like in Python pandas, the Series and the DataFrame."
• dataframe-js
  • "DataFrame-js provides an immutable data structure for javascript and datascience, the DataFrame, which allows to work on rows and columns with a sql and functional programming inspired api."
• data-forge
  • Seen in Ashley Davis" answer
  • "JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ."
  • Note the old data-forge JS repository is no longer maintained; now a new repository uses Typescript
• jsdataframe
  • "Jsdataframe is a JavaScript data wrangling library inspired by data frame functionality in R and Python Pandas."
• dataframe
  • "explore data by grouping and reducing."

Then after coming to this question, checking other answers here and doing more searching, I found options like:

• Apache Arrow in JS
  • Thanks to user Back2Basics suggestion:
  • "Apache Arrow is a columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. Apache Arrow is the emerging standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...)"
• Observable
  • At first glance, seems like a JS alternative to the IPython/Jupyter "notebooks"
  • Observable"s page promises: "Reactive programming", a "Community", on a "Web Platform"
  • See 5 minute intro here
• portal.js (formerly recline; from Rufus" answer)
  • MAY BE OUTDATED: Does not use a "DataFrame" API
  • MAY BE OUTDATED: Instead emphasizes its "Multiview" (the UI) API, (similar to jQuery/DOM model) which doesn"t require jQuery but does require a browser! More examples
  • MAY BE OUTDATED: Also emphasizes its MVC-ish architecture; including back-end stuff (i.e. database connections)
• js-data
  • Really more of an ORM! Most of its modules correspond to different data storage questions (js-data-mongodb, js-data-redis, js-data-cloud-datastore), sorting, filtering, etc.
  • On plus-side does work on Node.js as a first-priority; "Works in Node.js and in the Browser."
• miso (another suggestion from Rufus)
  • Impressive backers like Guardian and bocoup.
• AlaSQL
  • "AlaSQL" is an open source SQL database for Javascript with a strong focus on query speed and data source flexibility for both relational data and schemaless data. It works in your browser, Node.js, and Cordova."
• Some thought experiments:
  • "Scaling a DataFrame in Javascript" - Gary Sieling

Here are the criteria we used to consider the above choices

• General Criteria
  • Language (NodeJS vs browser JS vs Typescript)
  • Dependencies (i.e. if it uses an underlying library / AJAX/remote API"s)
  • Actively supported (active user-base, active source repository, etc)
  • Size/speed of JS library
• Panda"s criterias in its R comparison
  • Performance
  • Functionality/flexibility
  • Ease-of-use
• Similarity to Pandas / Dataframe API"s
  • Specifically hits on their main features
  • Data-science emphasis
  • Built-in visualization functions
  • Demonstrated integration in combination with other tools like Jupyter (interactive notebooks), etc

Answer #8

For a very simple range in ES6:

let range = n => Array.from(Array(n).keys())

From bigOmega"s comment, this can be shortened using Spread syntax:

let range = n => [...Array(n).keys()]

Answer #9

conda-env now does this automatically (if pip was installed with conda).

You can see how this works by using the export tool used for migrating an environment:

conda env export -n <env-name> > environment.yml

The file will list both conda packages and pip packages:

name: stats
channels:
  - javascript
dependencies:
  - python=3.4
  - bokeh=0.9.2
  - numpy=1.9.*
  - nodejs=0.10.*
  - flask
  - pip:
    - Flask-Testing

If you"re looking to follow through with exporting the environment, move environment.yml to the new host machine and run:

conda env create -f path/to/environment.yml

Answer #10

A coroutine is a generator function that can both yield values and accept values from the outside. The benefit of using a coroutine is that we can pause the execution of a function and resume it later. In case of a network operation, it makes sense to pause the execution of a function while we"re waiting for the response. We can use the time to run some other functions.

A future is like the Promise objects from Javascript. It is like a placeholder for a value that will be materialized in the future. In the above-mentioned case, while waiting on network I/O, a function can give us a container, a promise that it will fill the container with the value when the operation completes. We hold on to the future object and when it"s fulfilled, we can call a method on it to retrieve the actual result.

Direct Answer: You don"t need ensure_future if you don"t need the results. They are good if you need the results or retrieve exceptions occurred.

Extra Credits: I would choose run_in_executor and pass an Executor instance to control the number of max workers.

Explanations and Sample codes

In the first example, you are using coroutines. The wait function takes a bunch of coroutines and combines them together. So wait() finishes when all the coroutines are exhausted (completed/finished returning all the values).

loop = get_event_loop() # 
loop.run_until_complete(wait(coros))

The run_until_complete method would make sure that the loop is alive until the execution is finished. Please notice how you are not getting the results of the async execution in this case.

In the second example, you are using the ensure_future function to wrap a coroutine and return a Task object which is a kind of Future. The coroutine is scheduled to be executed in the main event loop when you call ensure_future. The returned future/task object doesn"t yet have a value but over time, when the network operations finish, the future object will hold the result of the operation.

from asyncio import ensure_future

futures = []
for i in range(5):
    futures.append(ensure_future(foo(i)))

loop = get_event_loop()
loop.run_until_complete(wait(futures))

So in this example, we"re doing the same thing except we"re using futures instead of just using coroutines.

Let"s look at an example of how to use asyncio/coroutines/futures:

import asyncio


async def slow_operation():
    await asyncio.sleep(1)
    return "Future is done!"


def got_result(future):
    print(future.result())

    # We have result, so let"s stop
    loop.stop()


loop = asyncio.get_event_loop()
task = loop.create_task(slow_operation())
task.add_done_callback(got_result)

# We run forever
loop.run_forever()

Here, we have used the create_task method on the loop object. ensure_future would schedule the task in the main event loop. This method enables us to schedule a coroutine on a loop we choose.

We also see the concept of adding a callback using the add_done_callback method on the task object.

A Task is done when the coroutine returns a value, raises an exception or gets canceled. There are methods to check these incidents.

I have written some blog posts on these topics which might help:

• http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html
• http://masnun.com/2015/11/20/python-asyncio-future-task-and-the-event-loop.html
• http://masnun.com/2015/12/07/python-3-using-blocking-functions-or-codes-with-asyncio.html

Of course, you can find more details on the official manual: https://docs.python.org/3/library/asyncio.html