{"id":1435,"date":"2020-08-17T10:31:07","date_gmt":"2020-08-17T08:31:07","guid":{"rendered":"https:\/\/cwiok.pl\/?p=1435"},"modified":"2020-08-17T10:34:53","modified_gmt":"2020-08-17T08:34:53","slug":"monitor-ofert-dla-otomoto-pl-usluga-architektura","status":"publish","type":"post","link":"https:\/\/cwiok.pl\/index.php\/pl\/2020\/08\/17\/monitor-ofert-dla-otomoto-pl-usluga-architektura\/","title":{"rendered":"Monitor ofert dla otomoto.pl [Us\u0142uga + architektura]"},"content":{"rendered":"

Wiele os\u00f3b stwierdzi\u0142o, \u017ce to w jaki spos\u00f3b kupi\u0142em sw\u00f3j samoch\u00f3d jest bardzo interesuj\u0105cy, dlatego postanowi\u0142em stworzy\u0107 rozwi\u0105zanie dla ka\u017cdego! Dzi\u0119ki mojej us\u0142udze mo\u017cesz przez 7 dni 4 razy dziennie dostawa\u0107 na maila nowe oferty z otomoto.pl, kt\u00f3re pojawi\u0142y si\u0119 w mi\u0119dzyczasie.<\/p>\n

\"\"<\/a><\/p>\n

Ograniczenie jakie teraz zastosowa\u0142em polega na limicie stron, kt\u00f3re generuje twoje zapytanie. Wi\u0119c wybierz wszystkie filtry na otomoto.pl i sprawd\u017a ile stron ofert masz. Je\u015bli poni\u017cej 10, to skopiuj adres i wklej na gotowej stronie z rozwi\u0105zaniem. Jedna uwaga: maile nie wy\u015bwietlaj\u0105 si\u0119 poprawnie w programie Outlook.<\/p>\n

Znajdziesz j\u0105 tutaj<\/a>.<\/strong><\/h3>\n

Otodom.pl i gratka.pl s\u0105 w przygotowaniu.<\/p>\n

Tutaj zaczyna si\u0119 cz\u0119\u015b\u0107 techniczna.<\/strong><\/p>\n

Skrypt, kt\u00f3ry stworzy\u0142em na w\u0142asne potrzeby dzia\u0142a\u0142 w oparciu do RPI + Telegram. Nie jest to idealny zestaw, gdy stworzy\u0107 rozwi\u0105zanie dla wszystkich. Zdecydowa\u0142em si\u0119 na zamian\u0119 Telegrama na maila, a RPI na Azure. Chmura u\u0142atwi spraw\u0119, poniewa\u017c nie b\u0119d\u0119 musia\u0142 otwiera\u0107 RPI na \u015bwiat, ani martwi\u0107 si\u0119 do sta\u0142y dost\u0119p do internetu dla RPI. Minusem jest zdecydowanie fakt, \u017ce p\u0142ac\u0119 za to z w\u0142asnej kieszeni \u2013 st\u0105d wszystkie ograniczenia.<\/p>\n

Ca\u0142e rozwi\u0105zanie opiera si\u0119 o 3 us\u0142ugi na Azure:<\/p>\n

    \n
  1. \n
    Blob storage \u2013 funkcjonuje jako baza danych. Wiem, \u017ce nie jest to idealne rozwi\u0105zanie, ale jest tanie i mam gotowe kawa\u0142ki Pythona do obs\u0142ugi.<\/div>\n<\/li>\n
  2. \n
    Logic App \u2013 trzy aplikacje, kt\u00f3re zarz\u0105dzaj\u0105 procesem rejestracji, a tak\u017ce wyzwalaj\u0105 pobieranie ofert<\/div>\n<\/li>\n
  3. \n
    Function \u2013 us\u0142uga, kt\u00f3ra pozwala na uruchomienie Pythona bez \u017cadnego serwera (serverless). Tym sposobem nie musz\u0119 stawia\u0107 dodatkowej maszyny.<\/div>\n<\/li>\n<\/ol>\n

    Korzystaj\u0105c z tych serwis\u00f3w stworzy\u0142em cztery oddzielne (ale wsp\u00f3\u0142pracuj\u0105ce) procesy:<\/p>\n

      \n
    1. \n
      Rejestracja adresu mailowego<\/div>\n<\/li>\n
    2. \n
      Potwierdzenie adresu mailowego<\/div>\n<\/li>\n
    3. \n
      Cykliczne wysy\u0142anie ofert otomoto.pl<\/div>\n<\/li>\n
    4. \n
      Usuni\u0119cie subskrypcji<\/div>\n<\/li>\n<\/ol>\n

      Poni\u017cej znajduj\u0105 si\u0119 szczeg\u00f3\u0142y ka\u017cdego z proces\u00f3w, a tak\u017ce kawa\u0142ki kodu.<\/p>\n

      1. Rejestracja adresu mailowego<\/p>\n

      \"Diag1PL\"<\/a><\/p>\n

      Widok logic app:<\/p>\n

      \"image\"<\/a><\/p>\n

      Proces rejestracji dzia\u0142a w bardzo prosty spos\u00f3b. U\u017cytkownik rejestruje si\u0119 za pomoc\u0105 formularza, kt\u00f3ry generuje zapytanie GET trafiaj\u0105ce do logic app. Zapytanie GET przekazuje email u\u017cytkownika i jego URL z otomoto. URL jest sprawdzany przez funkcj\u0119 GetPages (kod poni\u017cej), kt\u00f3ra sprawdza ile stron ma zapytanie. Je\u015bli ma wi\u0119cej ni\u017c 10 stron, u\u017cytkownik dostaje stosown\u0105 odpowied\u017a. Je\u015bli ma mniej ni\u017c 10 stron to s\u0105 tworzone pliki w blob storage, zostaje nadany id dla maila, wys\u0142any email z linkiem do potwierdzenia i pokazana odpowied\u017a potwierdzaj\u0105ca. Blob storage zawiera polityk\u0119, kt\u00f3ra usunie pliki (zapytanie i email) po 2 dniach. Je\u015bli u\u017cytkownik spr\u00f3buje klikn\u0105\u0107 z link po 2 dniach, proces rejestracji nie powiedzie si\u0119.<\/p>\n

      Klasa OtomotoScrapper, kt\u00f3ra zawiera wszystkie potrzebne metody:<\/p>\n

      import requests\r\nfrom lxml import html\r\nimport os\r\nimport json\r\nimport time\r\n\r\nimport datetime\r\n\r\n\r\nclass OtomotoScrapper:\r\n    \r\n    \r\n    def __init__(self, url, previous_offers):\r\n        self.headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n        self.url = url\r\n        self.previous_offers = previous_offers\r\n        self.new_offers = {}\r\n        self.no_of_pages = self.get_number_of_pages()\r\n        self.html_template =  \"\"\"\r\n\tSuper long html_template was here.\r\n    \"\"\"\r\n        \r\n\r\n        \r\n\r\n    def get_number_of_pages(self):\r\n        #This function will just retrieve the maximum number of pages on the website. This is used when iterating through n pages\r\n\r\n        url = self.url\r\n        \r\n        request = requests.get(url, headers = self.headers)\r\n        tree = html.fromstring(request.text)\r\n        xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'\r\n        max_page= tree.xpath('\/\/ul[@class=\"om-pager rel\"]\/li[last()-1]\/a\/span\/text()')\r\n        offers = tree.xpath(xpath_offer_details)\r\n        if not max_page and offers:\r\n            return 1\r\n        elif max_page:\r\n            max_page = max_page[0].strip()\r\n            #print(max_page)\r\n            return int(max_page)\r\n        else:\r\n            return 0\r\n\r\n\r\n    def get_offers(self, n):\r\n        url = str(self.url) +\"&page=\"+ str(n)\r\n        request = requests.get(url, headers = self.headers)\r\n        tree = html.fromstring(request.text)\r\n\r\n        xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'#\/\/text()\r\n        xpath_url = '\/\/div[@class=\"offers list\"]\/article\/@data-href'#\/\/text()\r\n        \r\n        offer_details = tree.xpath(xpath_offer_details)\r\n        list_of_urls = tree.xpath(xpath_url)\r\n        #print(list_of_urls)\r\n        for i, detail in enumerate(offer_details):\r\n            try:\r\n                if not list_of_urls[i] in self.previous_offers: #check if URLs was present before, if not download all the details\r\n\r\n                    self.previous_offers[list_of_urls[i]] = self.get_single_offer(detail)\r\n                    self.new_offers[list_of_urls[i]] = self.get_single_offer(detail)\r\n                    \r\n                    #VIN and Phone require seperate logic\r\n                    offer_id = list_of_urls[i].split(\"-ID\")[1].split(\".html\")[0]\r\n\r\n            except Exception as e:\r\n                print(e)\r\n                print(\"sss\")\r\n\r\n\r\n    def get_single_offer(self,html_element):\r\n        #This function will enter html_element and retrieve all offer details basing on xpath\r\n        single_offer_details = {}\r\n        single_offer_details['url'] = html_element.xpath('@data-href')[0]\r\n\r\n        single_offer_details['name'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__title\"]\/h2\/a')[0].text_content().strip()\r\n        single_offer_details['subtitle'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__title\"]\/h3')[0].text_content().strip()\r\n        single_offer_details['price'] = \" \".join(html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__price\"]\/div\/div\/span')[0].text_content().strip().split())\r\n        single_offer_details['foto'] = html_element.xpath('div[@class=\"offer-item__photo  ds-photo-container\"]\/a\/img\/@data-srcset')[0].split(';s=')[0]\r\n        \r\n        single_offer_details['offer_details'] =  html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/*[@class=\"ds-params-block\"]\/*[@class=\"ds-param\"]\/span\/text()')\r\n        single_offer_details['details_string'] = ' \u2022 '.join(single_offer_details['offer_details'])\r\n        return single_offer_details\r\n\r\n    \r\n    def get_everything(self):\r\n\r\n        #This function iterates through all pages, saving everything into globabl variable previous_offers that will be saves to json.\r\n        for i in range(1,self.get_number_of_pages()+1):\r\n            self.get_offers(i)\r\n            \r\n        return self.new_offers\r\n    \r\n\r\n\r\n    def get_vin_and_phone(self, id):\r\n        #Digging in website's code let me discover that Vin and Phone number are available under those URLs without any additional authentication\r\n        vin_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/vin\/\"\r\n        phone_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/contact\/multi_phone\/{}\/0\"\r\n\r\n        request = requests.get(vin_url+id)\r\n\r\n    \r\n        vin = request.text.replace(\"\\\"\",\"\")\r\n        request = requests.get(phone_url.format(id))\r\n\r\n        \r\n        phone = json.loads(request.text)[\"value\"].replace(\" \",\"\")\r\n\r\n        return vin, phone\r\n\r\n    def create_html(self):\r\n        offer_html = []\r\n        \r\n        for key in self.new_offers:\r\n\r\n            offer_html.append(self.html_template.format(**self.new_offers[key]))\r\n            \r\n        return ''.join(offer_html)\r\n\r\n\r\n\r\n\r\n#get_everything()\r\n#print(get_number_of_pages())\r\nif __name__ == \"__main__\":\r\n    \r\n    previos_offers={}\r\n    url = \"https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2017\/?search%5Bfilter_enum_generation%5D%5B0%5D=gen-ii-2016&search%5Bfilter_float_year%3Ato%5D=2018&search%5Bfilter_float_mileage%3Ato%5D=55000&search%5Bfilter_float_engine_power%3Afrom%5D=160&search%5Bfilter_enum_gearbox%5D%5B0%5D=automatic&search%5Bfilter_enum_gearbox%5D%5B1%5D=cvt&search%5Bfilter_enum_gearbox%5D%5B2%5D=dual-clutch&search%5Bfilter_enum_gearbox%5D%5B3%5D=semi-automatic&search%5Bfilter_enum_gearbox%5D%5B4%5D=automatic-stepless-sequential&search%5Bfilter_enum_gearbox%5D%5B5%5D=automatic-stepless&search%5Bfilter_enum_gearbox%5D%5B6%5D=automatic-sequential&search%5Bfilter_enum_gearbox%5D%5B7%5D=automated-manual&search%5Bfilter_enum_gearbox%5D%5B8%5D=direct-no-gearbox&search%5Bfilter_enum_country_origin%5D%5B0%5D=pl&search%5Border%5D=created_at%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=\"\r\n    scrapper = OtomotoScrapper(url, previos_offers)\r\n    scrapper.get_everything()\r\n    print(scrapper.create_html())\r\n\r\n\r\n\r\n    \r\n\r\n<\/pre>\n
      Funkcja GetPages:<\/p>\n
      import logging\r\nfrom ..shared_code import ScrappyScrapper\r\n\r\nimport azure.functions as func\r\nfrom lxml import html\r\nimport requests\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n        \r\n            \r\n        try: \r\n            previos_offers={}\r\n            scrapper = ScrappyScrapper.OtomotoScrapper(name, previos_offers)\r\n            return func.HttpResponse(str(scrapper.get_number_of_pages())) \r\n\r\n        except Exception as e:\r\n            return func.HttpResponse(e)\r\n        \r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n<\/pre>\n
       <\/p>\n
      2. Potwierdzenie adresu mailowego:<\/p>\n
      \"Diag2PL\"<\/a><\/p>\n
      Widok logic app:<\/p>\n
      \"image\"<\/a><\/p>\n
      Gdy u\u017cytkownik kliknie w link w emailu, nast\u0119puje sprawdzenie czy link z maila zawiera id takie samo jak to w blob storage. To zapobiegnie (mam nadziej\u0119) potwierdzeniu dowolnego maila bez dost\u0119pu do skrzynki. Kiedy to zostanie potwierdzone, zostaje wyliczany hash maila (za pomoc\u0105 funkcji Hash), \u017cebym mia\u0142 tylko jedno miejsce, z kt\u00f3rego musz\u0119 usun\u0105\u0107 pliki w przypadku usuni\u0119cia subskrypcji. Dodatkowo zostan\u0105 usuni\u0119te pliki u\u017cywane do rejestracji. Nast\u0119pnie zostaje wys\u0142any email z potwierdzeniem i linkiem do usuni\u0119cia subskrypcji, zostaje te\u017c wys\u0142ana odpowied\u017a z potwierdzeniem. Na koniec s\u0105 pobierane oferty na moment rejestracji (funkcja ScrapHttp).<\/p>\n
      Funkcja Hash:<\/p>\n
      import logging\r\nfrom hashlib import sha256\r\nimport azure.functions as func\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n        name = name.upper()\r\n        name = sha256(name.encode()).hexdigest()\r\n        return func.HttpResponse(name)\r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n<\/pre>\n
      Funkcja ScrapHttp:<\/p>\n
      import logging\r\nfrom ..shared_code import blob\r\nfrom ..shared_code import ScrappyScapper\r\nimport azure.functions as func\r\nimport json\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    \r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n\r\n        #download_data_for_email(name)\r\n        return func.HttpResponse(download_data_for_email(name))\r\n\r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n\r\ndef download_data_for_email(email):\r\n    po_file = blob.download_blob('offers\/'+email)\r\n    if po_file:\r\n        previos_offers = json.loads(po_file)\r\n    else:\r\n        previos_offers = {}\r\n\r\n    url = blob.download_blob('validatedqueries\/'+email)\r\n\r\n    scrapper = ScrappyScapper.OtomotoScrapper(url, previos_offers)\r\n\r\n    new_offers = scrapper.get_everything()\r\n    previos_offers = scrapper.previous_offers\r\n\r\n    blob.upload_to_blob(json.dumps(previos_offers),'offers\/'+email)\r\n    return scrapper.create_html()\r\n    #print(new_offers)\r\n<\/pre>\n
       <\/p>\n
      3. Cykliczne wysy\u0142anie ofert otomoto.pl<\/p>\n
      \"Diag3PL\"<\/a><\/p>\n
      Widok logic app:<\/p>\n
      \"image\"<\/a><\/p>\n
      Kilka razy dziennie aplikacja zostaje wyzwolona automatycznie. Dla ka\u017cdego hasha emaila z odpowiedniego \u201ckatalogu\u201d (w blob storage katalog to tylko cz\u0119\u015b\u0107 nazwy) s\u0105 pobierane nowe oferty. Je\u015bli nie ma nowych ofert nic si\u0119 nie dzieje. W przeciwnym wypadku dla hasha emaila jest odnajdowany email (ta informacja znajduje si\u0119 w blob storage), a nast\u0119pnie jest wysy\u0142any email.<\/p>\n
      4. Usuni\u0119cie subskrypcji<\/p>\n
      \"Diag4PL\"<\/a><\/p>\n
      Widok logic app:<\/p>\n
      \"image\"<\/a><\/p>\n
      Proces usuni\u0119cia emaila zaczyna si\u0119 od sprawdzenia czy email w og\u00f3le istnieje. Je\u015bli nie zostaje wy\u015bwietlona stosowna informacja. Je\u015bli email istnieje, sprawdzane jest czy id z zapytania pokrywa si\u0119 z id emaila. To zapobiegnie (mam nadziej\u0119) sytuacji, w kto\u015b usunie nieswojego maila. Kiedy to zostaje potwierdzone, liczony jest hash maila, zostaj\u0105 usuni\u0119te pliki, kt\u00f3re mapuj\u0105 hash i email. Pobrane oferty zostaj\u0105 skopiowane do archiwum i zostaje wys\u0142ana odpowied\u017a z potwierdzeniem.<\/p>\n
      I to tyle! M\u00f3j bud\u017cet to 5 USD miesi\u0119cznie, wi\u0119c korzystajcie! Ca\u0142y czas czuwam nad rejestrowanymi mailami, bo wiem, \u017ce da si\u0119 obej\u015b\u0107 moje limity. Je\u015bli kto\u015b b\u0119dzie przesadza\u0142, b\u0119d\u0119 banowa\u0142! Je\u015bli potrzebujecie cz\u0119stszych powiadomie\u0144, dajcie zna\u0107!<\/p>\n
      Startuj\u0119 z newsletterem. Je\u015bli chcesz dostawa\u0107 podobne zestawienia i moje analizy na maila – zapraszam.<\/p>\n
      \n
      <\/p>\n
      <\/div>\n<\/form>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
      Wiele os\u00f3b stwierdzi\u0142o, \u017ce to w jaki spos\u00f3b kupi\u0142em sw\u00f3j samoch\u00f3d jest bardzo interesuj\u0105cy, dlatego postanowi\u0142em stworzy\u0107 rozwi\u0105zanie dla ka\u017cdego! Dzi\u0119ki mojej us\u0142udze mo\u017cesz przez 7 dni 4 razy dziennie dostawa\u0107 na maila nowe oferty z otomoto.pl, kt\u00f3re pojawi\u0142y si\u0119 w mi\u0119dzyczasie.<\/p>\n
      \"\"<\/a><\/p>\n
      Read More<\/a><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,73,34],"tags":[],"class_list":["post-1435","post","type-post","status-publish","format-standard","hentry","category-azure-pl","category-projekty","category-python-pl"],"_links":{"self":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1435","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/comments?post=1435"}],"version-history":[{"count":18,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1435\/revisions"}],"predecessor-version":[{"id":1483,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1435\/revisions\/1483"}],"wp:attachment":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/media?parent=1435"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/categories?post=1435"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/tags?post=1435"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}