{"id":1453,"date":"2020-08-17T10:31:07","date_gmt":"2020-08-17T08:31:07","guid":{"rendered":"https:\/\/cwiok.pl\/?p=1453"},"modified":"2020-08-17T10:31:37","modified_gmt":"2020-08-17T08:31:37","slug":"offer-monitor-for-otomoto-pl-service-architecture","status":"publish","type":"post","link":"https:\/\/cwiok.pl\/index.php\/en\/2020\/08\/17\/offer-monitor-for-otomoto-pl-service-architecture\/","title":{"rendered":"Offer monitor for otomoto.pl [Service + Architecture]"},"content":{"rendered":"

Many people told me that the way I bought my car is very interesting. Therefore I decided to create a solution for everyone! Using my service you will be able to receive an email with new offers from otomoto.pl 4 times a day for 7 days.<\/p>\n

\"\"<\/a><\/p>\n

The only limitation in place is the number of pages you get in otomoto.pl Select all the filters (make, model etc.) and see how many pages this generates. If less than 10, copy the URL and paste in the form. One notice: emails seem to not work with Outlook.<\/p>\n

The form is available here<\/a>.<\/strong><\/h3>\n

Otodom.pl and gratka.pl are being worked on right now.<\/p>\n

This is where the technical part starts.<\/strong><\/p>\n

The script that I created previously was based on RPI + Telegram. This is not a perfect setup, if you want to create a solution available for everyone. I decided to change Telegram for a good ol’ email, and RPI for Azure. Cloud will be helpful here, as I will not have to open any ports or care about constant internet access. A big disadvantage is the fact that I will actually have to pay out of my pocket – hence the limitation.<\/p>\n

The whole solution is based on 3 Azure services:<\/p>\n

    \n
  1. \n
    Blob storage \u2013 functions as a database. I know that this is not perfect, but it’s super cheap and I have some Python scripts readily available.<\/div>\n<\/li>\n
  2. \n
    Logic App \u2013 three apps that manage registration process and downloading offers.<\/div>\n<\/li>\n
  3. \n
    Function \u2013 service that lets me run Python without any service (serverless thingy). This way I do not have to set up a new VM.<\/div>\n<\/li>\n<\/ol>\n

    Using those services, I created four separate (yet cooperating) processes:<\/p>\n

      \n
    1. \n
      Email registration<\/div>\n<\/li>\n
    2. \n
      Email confirmation<\/div>\n<\/li>\n
    3. \n
      Sending out new offers<\/div>\n<\/li>\n
    4. \n
      Subscription removal<\/div>\n<\/li>\n<\/ol>\n

      Below you can find details of each process and also pieces of code.<\/p>\n

      1. Email registration<\/p>\n

      \"Diag1ENG\"<\/a><\/p>\n

      Logic app view:<\/p>\n

      \"image\"<\/a><\/p>\n

      The registation process is rather easy to follow. A user registers through a form, which sends a GET request to the logic app. The requests passes user email and the URL from otomoto.pl. The URL is check using a function called GetPages (code below), to check for the number of pages. If it returns more than 10 pages, the users gets a proper response. If it returns less, files are created on blob storage, emails gets an ID, an email is sent with confirmation link and a proper response is shown. Blob storage has a policy, which will delete the files after 2 days (offer URL and email). If the user clicks on the confirmation link after 2 days, it will not work.<\/p>\n

      OtomotoScrapper class, which contains all the methods:<\/p>\n

      import requests\r\nfrom lxml import html\r\nimport os\r\nimport json\r\nimport time\r\n\r\nimport datetime\r\n\r\n\r\nclass OtomotoScrapper:\r\n    \r\n    \r\n    def __init__(self, url, previous_offers):\r\n        self.headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n        self.url = url\r\n        self.previous_offers = previous_offers\r\n        self.new_offers = {}\r\n        self.no_of_pages = self.get_number_of_pages()\r\n        self.html_template =  \"\"\"\r\n\tSuper long html_template was here.\r\n    \"\"\"\r\n        \r\n\r\n        \r\n\r\n    def get_number_of_pages(self):\r\n        #This function will just retrieve the maximum number of pages on the website. This is used when iterating through n pages\r\n\r\n        url = self.url\r\n        \r\n        request = requests.get(url, headers = self.headers)\r\n        tree = html.fromstring(request.text)\r\n        xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'\r\n        max_page= tree.xpath('\/\/ul[@class=\"om-pager rel\"]\/li[last()-1]\/a\/span\/text()')\r\n        offers = tree.xpath(xpath_offer_details)\r\n        if not max_page and offers:\r\n            return 1\r\n        elif max_page:\r\n            max_page = max_page[0].strip()\r\n            #print(max_page)\r\n            return int(max_page)\r\n        else:\r\n            return 0\r\n\r\n\r\n    def get_offers(self, n):\r\n        url = str(self.url) +\"&page=\"+ str(n)\r\n        request = requests.get(url, headers = self.headers)\r\n        tree = html.fromstring(request.text)\r\n\r\n        xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'#\/\/text()\r\n        xpath_url = '\/\/div[@class=\"offers list\"]\/article\/@data-href'#\/\/text()\r\n        \r\n        offer_details = tree.xpath(xpath_offer_details)\r\n        list_of_urls = tree.xpath(xpath_url)\r\n        #print(list_of_urls)\r\n        for i, detail in enumerate(offer_details):\r\n            try:\r\n                if not list_of_urls[i] in self.previous_offers: #check if URLs was present before, if not download all the details\r\n\r\n                    self.previous_offers[list_of_urls[i]] = self.get_single_offer(detail)\r\n                    self.new_offers[list_of_urls[i]] = self.get_single_offer(detail)\r\n                    \r\n                    #VIN and Phone require seperate logic\r\n                    offer_id = list_of_urls[i].split(\"-ID\")[1].split(\".html\")[0]\r\n\r\n            except Exception as e:\r\n                print(e)\r\n                print(\"sss\")\r\n\r\n\r\n    def get_single_offer(self,html_element):\r\n        #This function will enter html_element and retrieve all offer details basing on xpath\r\n        single_offer_details = {}\r\n        single_offer_details['url'] = html_element.xpath('@data-href')[0]\r\n\r\n        single_offer_details['name'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__title\"]\/h2\/a')[0].text_content().strip()\r\n        single_offer_details['subtitle'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__title\"]\/h3')[0].text_content().strip()\r\n        single_offer_details['price'] = \" \".join(html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__price\"]\/div\/div\/span')[0].text_content().strip().split())\r\n        single_offer_details['foto'] = html_element.xpath('div[@class=\"offer-item__photo  ds-photo-container\"]\/a\/img\/@data-srcset')[0].split(';s=')[0]\r\n        \r\n        single_offer_details['offer_details'] =  html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/*[@class=\"ds-params-block\"]\/*[@class=\"ds-param\"]\/span\/text()')\r\n        single_offer_details['details_string'] = ' \u2022 '.join(single_offer_details['offer_details'])\r\n        return single_offer_details\r\n\r\n    \r\n    def get_everything(self):\r\n\r\n        #This function iterates through all pages, saving everything into globabl variable previous_offers that will be saves to json.\r\n        for i in range(1,self.get_number_of_pages()+1):\r\n            self.get_offers(i)\r\n            \r\n        return self.new_offers\r\n    \r\n\r\n\r\n    def get_vin_and_phone(self, id):\r\n        #Digging in website's code let me discover that Vin and Phone number are available under those URLs without any additional authentication\r\n        vin_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/vin\/\"\r\n        phone_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/contact\/multi_phone\/{}\/0\"\r\n\r\n        request = requests.get(vin_url+id)\r\n\r\n    \r\n        vin = request.text.replace(\"\\\"\",\"\")\r\n        request = requests.get(phone_url.format(id))\r\n\r\n        \r\n        phone = json.loads(request.text)[\"value\"].replace(\" \",\"\")\r\n\r\n        return vin, phone\r\n\r\n    def create_html(self):\r\n        offer_html = []\r\n        \r\n        for key in self.new_offers:\r\n\r\n            offer_html.append(self.html_template.format(**self.new_offers[key]))\r\n            \r\n        return ''.join(offer_html)\r\n\r\n\r\n\r\n\r\n#get_everything()\r\n#print(get_number_of_pages())\r\nif __name__ == \"__main__\":\r\n    \r\n    previos_offers={}\r\n    url = \"https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2017\/?search%5Bfilter_enum_generation%5D%5B0%5D=gen-ii-2016&search%5Bfilter_float_year%3Ato%5D=2018&search%5Bfilter_float_mileage%3Ato%5D=55000&search%5Bfilter_float_engine_power%3Afrom%5D=160&search%5Bfilter_enum_gearbox%5D%5B0%5D=automatic&search%5Bfilter_enum_gearbox%5D%5B1%5D=cvt&search%5Bfilter_enum_gearbox%5D%5B2%5D=dual-clutch&search%5Bfilter_enum_gearbox%5D%5B3%5D=semi-automatic&search%5Bfilter_enum_gearbox%5D%5B4%5D=automatic-stepless-sequential&search%5Bfilter_enum_gearbox%5D%5B5%5D=automatic-stepless&search%5Bfilter_enum_gearbox%5D%5B6%5D=automatic-sequential&search%5Bfilter_enum_gearbox%5D%5B7%5D=automated-manual&search%5Bfilter_enum_gearbox%5D%5B8%5D=direct-no-gearbox&search%5Bfilter_enum_country_origin%5D%5B0%5D=pl&search%5Border%5D=created_at%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=\"\r\n    scrapper = OtomotoScrapper(url, previos_offers)\r\n    scrapper.get_everything()\r\n    print(scrapper.create_html())\r\n\r\n\r\n\r\n    \r\n\r\n<\/pre>\n
      GetPages:<\/p>\n
      import logging\r\nfrom ..shared_code import ScrappyScrapper\r\n\r\nimport azure.functions as func\r\nfrom lxml import html\r\nimport requests\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n        \r\n            \r\n        try: \r\n            previos_offers={}\r\n            scrapper = ScrappyScrapper.OtomotoScrapper(name, previos_offers)\r\n            return func.HttpResponse(str(scrapper.get_number_of_pages())) \r\n\r\n        except Exception as e:\r\n            return func.HttpResponse(e)\r\n        \r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n<\/pre>\n
       <\/p>\n
      2. Email confirmation:<\/p>\n
      \"Diag2PL\"<\/a><\/p>\n
      Logic app view:<\/p>\n
      \"image\"<\/a><\/p>\n
      When a users click the confirmation link, a check happens to validate email’s id. This (hopefully) will make it impossible to confirm an email address without an actual access to it. Afterwards the process calculates SHA256 hash of the email (using Hash function), this is done so that I have only one place to delete from. The files used for the confirmation are then deleted. A welcome email with subscription removal link is sent. Later a response is shown to confirm the process. The last step is downloading all the existing offers for the moment of confirmation (ScrapHttp).<\/p>\n
      Hash:<\/p>\n
      import logging\r\nfrom hashlib import sha256\r\nimport azure.functions as func\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n        name = name.upper()\r\n        name = sha256(name.encode()).hexdigest()\r\n        return func.HttpResponse(name)\r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n<\/pre>\n
      ScrapHttp:<\/p>\n
      import logging\r\nfrom ..shared_code import blob\r\nfrom ..shared_code import ScrappyScapper\r\nimport azure.functions as func\r\nimport json\r\n\r\n\r\ndef main(req: func.HttpRequest) -> func.HttpResponse:\r\n    logging.info('Python HTTP trigger function processed a request.')\r\n\r\n    \r\n    name = req.params.get('name')\r\n    if not name:\r\n        try:\r\n            req_body = req.get_json()\r\n        except ValueError:\r\n            pass\r\n        else:\r\n            name = req_body.get('name')\r\n\r\n    if name:\r\n\r\n        #download_data_for_email(name)\r\n        return func.HttpResponse(download_data_for_email(name))\r\n\r\n    else:\r\n        return func.HttpResponse(\r\n             \"Please pass a name on the query string or in the request body\",\r\n             status_code=400\r\n        )\r\n\r\ndef download_data_for_email(email):\r\n    po_file = blob.download_blob('offers\/'+email)\r\n    if po_file:\r\n        previos_offers = json.loads(po_file)\r\n    else:\r\n        previos_offers = {}\r\n\r\n    url = blob.download_blob('validatedqueries\/'+email)\r\n\r\n    scrapper = ScrappyScapper.OtomotoScrapper(url, previos_offers)\r\n\r\n    new_offers = scrapper.get_everything()\r\n    previos_offers = scrapper.previous_offers\r\n\r\n    blob.upload_to_blob(json.dumps(previos_offers),'offers\/'+email)\r\n    return scrapper.create_html()\r\n    #print(new_offers)\r\n<\/pre>\n
       <\/p>\n
      3. Sending out new offers<\/p>\n
      \"Diag3PL\"<\/a><\/p>\n
      Logic app view:<\/p>\n
      \"image\"<\/a><\/p>\n
      A few times a day this app is triggered. For each email hash from a selected directory (directories in blob storage are just part of the name) the offers are downloaded. If there are no new offers, nothing happens. If there are some new offers, for the selected hash an email is being found (this information is inside blob storage) and an emails is sent.<\/p>\n
      4. Subscription removal<\/p>\n
      \"Diag4PL\"<\/a><\/p>\n
      Logic app view:<\/p>\n
      \"image\"<\/a><\/p>\n
      This process starts with a check if the email even exists. If not, a proper response is shown. If the email indeed exists, another check happens to confirm its ID. This will prevent deleting random emails. After this is confirmed, the hash is calculated, and all the files (hash-email pair, email id) are deleted. Downloaded offers are then copied to the archive. Afterwards a response with confirmation is shown.<\/p>\n
      And that is it! My budget is 5 USD per month, so use it! I monitor closely all the emails, as I know that you can probably bypass the system. If someone pushes it to the limits, I will ban! If you need more frequent notifications, let me know!<\/p>\n
      I start with a newsletter. If you want to be notified of such great pieces of work – subscribe below.<\/p>\n
      \n
      <\/p>\n
      <\/div>\n<\/form>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
      Many people told me that the way I bought my car is very interesting. Therefore I decided to create a solution for everyone! Using my service you will be able to receive an email with new offers from otomoto.pl 4 times a day for 7 days.<\/p>\n
      \"\"<\/a><\/p>\n
      Read More<\/a><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[75],"tags":[],"class_list":["post-1453","post","type-post","status-publish","format-standard","hentry","category-projects"],"_links":{"self":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/comments?post=1453"}],"version-history":[{"count":14,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1453\/revisions"}],"predecessor-version":[{"id":1482,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1453\/revisions\/1482"}],"wp:attachment":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/media?parent=1453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/categories?post=1453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/tags?post=1453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}