011.Test_requests&requests_futures.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# http://docs.python-requests.org/zh_CN/latest/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Blocking Or Non-Blocking?\n",
    "With the default Transport Adapter in place, \n",
    "Requests does not provide any kind of non-blocking IO. \n",
    "The Response.content property will block until the entire response has been downloaded. \n",
    "If you require more granularity, the streaming features of the library (see Streaming Requests) \n",
    "allow you to retrieve smaller quantities of the response at a time. \n",
    "However, these calls will still block.\n",
    "If you are concerned about the use of blocking IO, \n",
    "there are lots of projects out there that combine Requests with one of Python’s asynchronicity frameworks. \n",
    "Some excellent examples are requests-threads, grequests, and requests-futures.\n",
    "\n",
    "阻塞和非阻塞\n",
    "使用默认的传输适配器，Requests 不提供任何形式的非阻塞 IO。\n",
    "Response.content 属性会阻塞，直到整个响应下载完成。\n",
    "如果你需要更多精细控制，该库的数据流功能（见流式请求）允许你每次接受少量的一部分响应，不过这些调用依然是阻塞式的。\n",
    "如果你对于阻塞式 IO 有所顾虑，还有很多项目可以供你使用，它们结合了 Requests 和 Python 的某个异步框架。\n",
    "典型的优秀例子是 grequests 和 requests-futures。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.25 ms, sys: 885 µs, total: 6.13 ms\n",
      "Wall time: 28.9 ms\n",
      "CPU times: user 3.03 ms, sys: 4 µs, total: 3.04 ms\n",
      "Wall time: 38.6 ms\n",
      "CPU times: user 2.43 ms, sys: 0 ns, total: 2.43 ms\n",
      "Wall time: 33.2 ms\n",
      "CPU times: user 2.37 ms, sys: 29 µs, total: 2.4 ms\n",
      "Wall time: 41.2 ms\n",
      "CPU times: user 2.52 ms, sys: 0 ns, total: 2.52 ms\n",
      "Wall time: 39.2 ms\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "def f():\n",
    "    sess = requests.session()\n",
    "    p = sess.get('http://www.baidu.com')\n",
    "    sess.close()\n",
    "    \n",
    "for i in range(N):\n",
    "    %time f()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.68 ms, sys: 1.81 ms, total: 3.49 ms\n",
      "Wall time: 28.4 ms\n",
      "CPU times: user 2.79 ms, sys: 0 ns, total: 2.79 ms\n",
      "Wall time: 15.4 ms\n",
      "CPU times: user 2.46 ms, sys: 68 µs, total: 2.53 ms\n",
      "Wall time: 15.8 ms\n",
      "CPU times: user 2.36 ms, sys: 132 µs, total: 2.49 ms\n",
      "Wall time: 15.7 ms\n",
      "CPU times: user 2.17 ms, sys: 127 µs, total: 2.3 ms\n",
      "Wall time: 15.3 ms\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "sess = requests.session()\n",
    "for i in range(N):\n",
    "    %time p = sess.get('http://www.baidu.com')\n",
    "\n",
    "sess.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.58 ms, sys: 83 µs, total: 3.66 ms\n",
      "Wall time: 26.7 ms\n",
      "CPU times: user 2.35 ms, sys: 48 µs, total: 2.39 ms\n",
      "Wall time: 15.2 ms\n",
      "CPU times: user 2.21 ms, sys: 451 µs, total: 2.66 ms\n",
      "Wall time: 16.4 ms\n",
      "CPU times: user 2.5 ms, sys: 108 µs, total: 2.61 ms\n",
      "Wall time: 15.5 ms\n",
      "CPU times: user 2.58 ms, sys: 0 ns, total: 2.58 ms\n",
      "Wall time: 15.8 ms\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "sess = requests.session()\n",
    "for i in range(N):\n",
    "    %time sess.get('http://www.baidu.com')\n",
    "\n",
    "sess.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 126 µs, sys: 18 µs, total: 144 µs\n",
      "Wall time: 149 µs\n",
      "CPU times: user 74 µs, sys: 0 ns, total: 74 µs\n",
      "Wall time: 77.2 µs\n",
      "CPU times: user 57 µs, sys: 9 µs, total: 66 µs\n",
      "Wall time: 69.9 µs\n",
      "CPU times: user 64 µs, sys: 0 ns, total: 64 µs\n",
      "Wall time: 67.7 µs\n",
      "CPU times: user 61 µs, sys: 0 ns, total: 61 µs\n",
      "Wall time: 65.1 µs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/soft/py3/lib/python3.6/site-packages/grequests.py:21: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016\n",
      "  curious_george.patch_all(thread=False, select=False)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import grequests\n",
    "tasks = []\n",
    "for i in range(N):\n",
    "    %time grequests.get('http://www.baidu.com')\n",
    "\n",
    "grequests.map(tasks, size=4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "By default a ThreadPoolExecutor is created with 2 workers. If you would like to adjust that value or share a executor across multiple sessions you can provide one to the FuturesSession constructor.\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from requests_futures.sessions import FuturesSession\n",
    "```\n",
    "session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))\n",
    "...\n",
    "```\n",
    "As a shortcut in case of just increasing workers number you can pass max_workers straight to the FuturesSession constructor:\n",
    "from requests_futures.sessions import FuturesSession\n",
    "```\n",
    "session = FuturesSession(max_workers=10)\n",
    "```\n",
    "FutureSession will use an existing session object if supplied:\n",
    "```\n",
    "from requests import session\n",
    "from requests_futures.sessions import FuturesSession\n",
    "my_session = session()\n",
    "future_session = FuturesSession(session=my_session)\n",
    "```\n",
    "That's it. The api of requests.Session is preserved without any modifications beyond returning a Future rather than Response. As with all futures exceptions are shifted (thrown) to the future.result() call so try/except blocks should be moved there.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.56 ms, sys: 945 µs, total: 3.51 ms\n",
      "Wall time: 3.2 ms\n",
      "CPU times: user 1.24 ms, sys: 32 µs, total: 1.27 ms\n",
      "Wall time: 1.18 ms\n",
      "CPU times: user 44 µs, sys: 0 ns, total: 44 µs\n",
      "Wall time: 49.6 µs\n",
      "CPU times: user 28 µs, sys: 0 ns, total: 28 µs\n",
      "Wall time: 31.5 µs\n",
      "CPU times: user 23 µs, sys: 4 µs, total: 27 µs\n",
      "Wall time: 30.3 µs\n"
     ]
    }
   ],
   "source": [
    "from requests import session\n",
    "from requests_futures.sessions import FuturesSession\n",
    "my_session = session()\n",
    "sess = FuturesSession(session = my_session)\n",
    "\n",
    "for i in range(N):\n",
    "    %time sess.get('http://www.baidu.com')\n",
    "\n",
    "sess.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}