{"id":1382,"date":"2020-07-04T19:28:28","date_gmt":"2020-07-04T17:28:28","guid":{"rendered":"https:\/\/cwiok.pl\/?p=1382"},"modified":"2020-07-04T19:36:15","modified_gmt":"2020-07-04T17:36:15","slug":"kupowanie-samochodu-za-pomoca-pythona-rpi-i-telegrama","status":"publish","type":"post","link":"https:\/\/cwiok.pl\/index.php\/pl\/2020\/07\/04\/kupowanie-samochodu-za-pomoca-pythona-rpi-i-telegrama\/","title":{"rendered":"Kupowanie samochodu za pomoc\u0105 Pythona, RPI i Telegrama"},"content":{"rendered":"

Jaki\u015b czas temu zacz\u0105\u0142em rozgl\u0105da\u0107 si\u0119 za samochodem. Szybko znu\u017cy\u0142o mnie przegl\u0105danie ofert na np. otomoto.pl, wi\u0119c postanowi\u0142em zautomatyzowa\u0107 ca\u0142y proces i u\u0142atwi\u0107 sobie \u017cycie.<\/p>\n

\"\"<\/a><\/p>\n

Pomys\u0142 na ca\u0142y system jest bardzo prosty. Mam kod w Pythonie, kt\u00f3ry pobiera nowe oferty z otomoto.pl (dla wybranego przeze mnie samochodu). Kod jest uruchamiany co 10 minut na RPI za pomoc\u0105 cron joba, a nast\u0119pnie wysy\u0142a do mnie wszystkie nowe oferty na kanale na Telegramie. Tym sposobem nie musz\u0119 tworzy\u0107 nowego systemu powiadomie\u0144, a mam informacj\u0119 od razu na i kom\u00f3rce, i komputerze.<\/p>\n

Pierwszym krokiem jest naturalnie pobranie wszystkich ofert. Podzieli\u0142em to na 3 funkcje: pobranie listy ofert na jednej stronie, pobranie szczeg\u00f3\u0142\u00f3w jednej oferty, a potem pobranie wszystkich stron. Wszystkie oferty b\u0119d\u0105 zapisywane w pliku w formacie JSON, kt\u00f3ry b\u0119dzie wczytywany na samym pocz\u0105tku kodu. Tym sposobem b\u0119d\u0119 m\u00f3g\u0142 sprawdzi\u0107, kt\u00f3re oferty s\u0105 nowe, a kt\u00f3re stare.<\/p>\n

Prac\u0119 rozpoczynam od analizy kodu otomoto.pl, tutaj przyda si\u0119 narz\u0119dzie do inspekcji kodu (Ctrl + Shift + I) i podgl\u0105du sieci (Ctrl + Shift + E). Szybko zauwa\u017cam, \u017ce wszystkie oferty znajduj\u0105 si\u0119 w elemencie “article”:<\/p>\n

\"clip_image002[6]\"<\/a><\/p>\n

Dlatego w pierwszej funkcji, jak\u0105 stworz\u0119 podaj\u0119 w\u0142a\u015bnie “article” nawiguj\u0105c za pomoc\u0105 xpath. W parametrze URL s\u0105 wszystkie filtry z otomoto.pl, na jakich mi zale\u017cy. Nast\u0119pnie pobieram wszystkie kawa\u0142ki kodu pod “article” i sprawdzam jaki jest adres URL ka\u017cdego z nich. Je\u015bli nie ma go w pliku ze starymi ofertami, pobieram szczeg\u00f3\u0142y i wysy\u0142am powiadomienie.<\/p>\n

def get_offers(n):\r\n\r\n#This URL is what I was looking for, but you may change it as you need\r\n\r\n    url = 'https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2013\/?search%5Bfilter_float_year%3Ato%5D=2016&search%5Bfilter_float_mileage%3Ato%5D=100000&search%5Bfilter_enum_fuel_type%5D%5B0%5D=petrol&search%5Border%5D=created_at_first%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=&page='+str(n)\r\n\r\n    headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n\r\n    request = requests.get(url, headers = headers)\r\n\r\n    tree = html.fromstring(request.text)\r\n\r\n    xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'#\/\/text()\r\n\r\n    xpath_url = '\/\/div[@class=\"offers list\"]\/article\/@data-href'#\/\/text()\r\n\r\n    offer_details = tree.xpath(xpath_offer_details)\r\n\r\n    list_of_urls = tree.xpath(xpath_url)\r\n\r\nfor i, detail in enumerate(offer_details):\r\n\r\ntry:\r\n\r\nif not list_of_urls[i] in previous_offers: #check if URLs was present before, if not download all the details\r\n\r\n                previous_offers[list_of_urls[i]] = get_single_offer(detail)\r\n\r\n                sendTelegram(list_of_urls[i])\r\n\r\n#VIN and Phone require seperate logic\r\n\r\n                offer_id = list_of_urls[i].split(\"-ID\")[1].split(\"html\")[0]\r\n\r\n                sendTelegram(\"VIN: \"+get_vin_and_phone(offer_id)[0])\r\n\r\n                sendTelegram(\"Phone: \" + get_vin_and_phone(offer_id)[1])\r\n\r\nexcept Exception as e:\r\n\r\nprint(e)\r\n<\/pre>\n
 <\/p>\n
Aby pobra\u0107 VIN i Telefon musia\u0142em sprawdzi\u0107 ruch, jaki jest generowany przez stron\u0119. Tym sposobem odkry\u0142em pod jakim adresem te informacje si\u0119 znajduj\u0105. Nie b\u0119d\u0119 musia\u0142 przechodzi\u0107 przez ReCaptcha ani uruchamia\u0107 JavaScript.<\/p>\n
\"clip_image003\"<\/a><\/p>\n
Na tej podstawie napisa\u0142em funkcj\u0119:<\/p>\n
def get_vin_and_phone(id):\r\n\r\n#Digging in website's code let me discover that Vin and Phone number are available under those URLs without any additional authentication\r\n\r\n    vin_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/vin\/\"\r\n\r\n    phone_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/contact\/multi_phone\/{}\/0\"\r\n\r\n    request = requests.get(vin_url+id)\r\n\r\n    vin = request.text.replace(\"\\\"\",\"\")\r\n\r\n    request = requests.get(phone_url.format(id))\r\n\r\n    phone = json.loads(request.text)[\"value\"].replace(\" \",\"\")\r\n\r\nreturn vin, phone\r\n<\/pre>\n
 <\/p>\n
 <\/p>\n
Aby pobra\u0107 VIN lub telefon wystarczy wklei\u0107 w przegl\u0105dark\u0119 https:\/\/www.otomoto.pl\/ajax\/misc\/vin\/<\/a> + ID oferty, kt\u00f3ra znajduje si\u0119 na ko\u0144cu adresu ka\u017cdej oferty.<\/p>\n
Kolejna funkcja pobiera szczeg\u00f3\u0142y ka\u017cdej oferty zapisanej pod article. Tutaj r\u00f3wnie\u017c bardzo przydaje si\u0119 inspekcja kodu. Na czerwono zaznaczy\u0142em elementy, kt\u00f3re mnie interesuj\u0105:<\/p>\n
\"image\"<\/a><\/p>\n
Tutaj r\u00f3wnie\u017c nawiguj\u0105c za pomoc\u0105 Xpath, pobieram informacje, kt\u00f3re s\u0105 dla mnie istotne. Dodatkowo, maj\u0105c URL zdj\u0119cia samochodu, pobieram to zdj\u0119cie na dysk i zapisuje jako image.jpeg. To zdj\u0119cie zostanie mi wys\u0142ane przez Telegram razem ze wszystkimi szczeg\u00f3\u0142ami oferty.<\/p>\n
def get_single_offer(html_element):\r\n\r\n#This function will enter html_element and retrieve all offer details basing on xpath\r\n\r\n    single_offer_details = {}\r\n\r\n    single_offer_details['price'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__price\"]\/div\/div\/span\/span')[0].text_content().strip()\r\n\r\n    single_offer_details['foto'] = html_element.xpath('div[@class=\"offer-item__photo  ds-photo-container\"]\/a\/img\/@data-srcset')[0].split(';s=')[0]\r\n\r\n    single_offer_details['offer_details'] =  html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/*[@class=\"ds-params-block\"]\/*[@class=\"ds-param\"]\/span\/text()')\r\n\r\n    urllib.request.urlretrieve(single_offer_details['foto'], \"\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/image.jpeg\") #This will save photo from URL and save it locally. This will enable me to add this to my telegram\r\n\r\n    sendTelegram('Nowy Tiguan, Price: '+ single_offer_details['price']+', Details: ' + ', '.join(offer_details))\r\n\r\n    sendPhoto()\r\n\r\nreturn single_offer_details\r\n<\/pre>\n
 <\/p>\n
Aby pobra\u0107 wiele stron musz\u0119 mie\u0107 informacj\u0119 ile ich jest. Dlatego za pomoc\u0105 inspektora sprawdzam, gdzie znajduje si\u0119 to w kodzie HTML i pobieram za pomoc\u0105 funkcji:<\/p>\n
def get_number_of_pages():\r\n\r\n#This function will just retrieve the maximum number of pages on the website. This is used when iterating through n pages\r\n\r\n    url = 'https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2016\/?search%5Bfilter_float_year%3Ato%5D=2018&search%5Bfilter_float_mileage%3Ato%5D=60000&search%5Bfilter_enum_fuel_type%5D%5B0%5D=petrol&search%5Border%5D=created_at_first%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=&page=1'\r\n\r\n    headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n\r\n    request = requests.get(url, headers = headers)\r\n\r\n    tree = html.fromstring(request.text)\r\n\r\n    max_page= tree.xpath('\/\/ul[@class=\"om-pager rel\"]\/li[last()-1]\/a\/span\/text()')[0].strip()\r\n\r\nreturn int(max_page)\r\n<\/pre>\n
 <\/p>\n
 <\/p>\n
Maj\u0105c ju\u017c te kawa\u0142ki kodu, mog\u0142em zacz\u0105\u0107 wysy\u0142k\u0119 poprzez Telegram. \u017beby mie\u0107 tak\u0105 mo\u017cliwo\u015b\u0107 musia\u0142em stworzy\u0107 Bota. Aby to zrobi\u0107 nale\u017cy rozpocz\u0105\u0107 rozmow\u0119 z BotFather<\/b>. Wystarczy napisa\u0107 \/newbot, poda\u0107 nazw\u0119 Bota, a w odpowiedzi dostaniemy token.<\/p>\n
\"clip_image005\"<\/a><\/p>\n
Korzystaj\u0105c z tego tokena mog\u0142em stworzy\u0107 dwie funkcje. Jedna do wysy\u0142ania wiadomo\u015bci na czacie, a druga do wysy\u0142ania zdj\u0119cia image.jpeg do czatu.<\/p>\n
def sendTelegram(message):\r\n\r\n    token = '<Your Bot Token>'\r\n\r\n    method = 'sendMessage'\r\n\r\n    url = 'https:\/\/api.telegram.org\/bot{0}\/{1}'.format(token, method)\r\n\r\ntry:\r\n\r\n        response = requests.post(url=url , data = {'chat_id':<Your Chat ID>,'text':message , 'attachments':[{}]}).json()\r\n\r\nprint(response)\r\n\r\nexcept Exception as e: print(e)\r\n\r\ndef sendPhoto():\r\n\r\n    token = '<Your Bot Token>'\r\n\r\n    method = 'sendPhoto'\r\n\r\n    data={'chat_id': <Your Chat ID>}\r\n\r\n    files = {'photo': (\"\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/image.jpeg\", open(\"\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/image.jpeg\",'rb'))}\r\n\r\ntry:\r\n\r\n        response = requests.post(url='https:\/\/api.telegram.org\/bot{0}\/{1}'.format(token, 'sendPhoto'),data=data, files=files).json()\r\n\r\nprint(response)\r\n\r\nexcept Exception as e: print(e)\r\n<\/pre>\n
 <\/p>\n
 <\/p>\n
Maj\u0105c ju\u017c gotowy kod, mog\u0142em zapisa\u0107 do na moim RPI, kt\u00f3ry jest stale uruchomiony. Tym sposobem jestem pewien, \u017ce skrypt zawsze b\u0119dzie dzia\u0142a\u0142. Aby kod odpala\u0142 si\u0119 automatycznie skorzysta\u0142em z us\u0142ugi CRON, kt\u00f3ra jest swoistym harmonogramem zada\u0144 na linux. Aby doda\u0107 nowe zadanie wpisuj\u0119:<\/p>\n
crontab -e<\/pre>\n
 <\/p>\n
A nast\u0119pnie dodaj\u0119 w pliku adres mojego skryptu:<\/p>\n
0,10,20,30,40,50 * * * * \/usr\/bin\/python3 \/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/tiguan.py<\/pre>\n
Taki zapis oznacza, \u017ce o dowolnego dnia, o dowolnej godzinie, kt\u00f3ra mi minuty 10, 20, 30 itp. skrypt zadzia\u0142a.<\/p>\n
I w\u0142a\u015bnie w ten spos\u00f3b uda\u0142o mi si\u0119 stworzy\u0107 automat, kt\u00f3ry powiadamia mnie o ciekawych ofertach. Przyk\u0142ad dzia\u0142ania:<\/p>\n
\"clip_image006\"<\/a><\/p>\n
Tym sposobem mog\u0119 by\u0107 pierwsz\u0105 osob\u0105, kt\u00f3ra dzwoni do sprzedaj\u0105cego i mie\u0107 przewag\u0119 pierwsze\u0144stwa.<\/p>\n
Poni\u017cej ca\u0142y kod. Zapraszam do komentowania.<\/p>\n
import requests\r\n\r\nfrom lxml import html\r\n\r\nimport os\r\n\r\nimport json\r\n\r\nimport time\r\n\r\nfrom telegram import sendTelegram\r\n\r\nfrom telegram import sendPhoto\r\n\r\nimport datetime\r\n\r\nimport urllib.request\r\n\r\n# All offers are saved in Json file, so at the beginning I load everything into variable, to check for URLs. I treat URLs as unique identifiers\r\n\r\nwith open('\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/tiguan.json') as json_file:\r\n\r\n    previous_offers = json.load(json_file)\r\n\r\n# This function will open n-th page and save all details of offers that are not present in tiguan.json\r\n\r\ndef get_offers(n):\r\n\r\n#This URL is what I was looking for, but you may change it as you need\r\n\r\n    url = 'https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2013\/?search%5Bfilter_float_year%3Ato%5D=2016&search%5Bfilter_float_mileage%3Ato%5D=100000&search%5Bfilter_enum_fuel_type%5D%5B0%5D=petrol&search%5Border%5D=created_at_first%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=&page='+str(n)\r\n\r\n    headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n\r\n    request = requests.get(url, headers = headers)\r\n\r\n    tree = html.fromstring(request.text)\r\n\r\n    xpath_offer_details = '\/\/div[@class=\"offers list\"]\/article'#\/\/text()\r\n\r\n    xpath_url = '\/\/div[@class=\"offers list\"]\/article\/@data-href'#\/\/text()\r\n\r\n    offer_details = tree.xpath(xpath_offer_details)\r\n\r\n    list_of_urls = tree.xpath(xpath_url)\r\n\r\nfor i, detail in enumerate(offer_details):\r\n\r\ntry:\r\n\r\nif not list_of_urls[i] in previous_offers: #check if URLs was present before, if not download all the details\r\n\r\n                previous_offers[list_of_urls[i]] = get_single_offer(detail)\r\n\r\n                sendTelegram(list_of_urls[i])\r\n\r\n#VIN and Phone require seperate logic\r\n\r\n                offer_id = list_of_urls[i].split(\"-ID\")[1].split(\".html\")[0]\r\n\r\nprint(offer_id)\r\n\r\n                sendTelegram(\"VIN: \"+str(get_vin_and_phone(offer_id)[0]))\r\n\r\n                sendTelegram(\"Phone: \" + str(get_vin_and_phone(offer_id)[1]))\r\n\r\nexcept Exception as e:\r\n\r\nprint(e)\r\n\r\nprint(\"sss\")\r\n\r\ndef get_single_offer(html_element):\r\n\r\n#This function will enter html_element and retrieve all offer details basing on xpath\r\n\r\n    single_offer_details = {}\r\n\r\n    single_offer_details['price'] = html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/div[@class=\"offer-item__price\"]\/div\/div\/span\/span')[0].text_content().strip()\r\n\r\n    single_offer_details['foto'] = html_element.xpath('div[@class=\"offer-item__photo  ds-photo-container\"]\/a\/img\/@data-srcset')[0].split(';s=')[0]\r\n\r\n    single_offer_details['offer_details'] =  html_element.xpath('div[@class=\"offer-item__content ds-details-container\"]\/*[@class=\"ds-params-block\"]\/*[@class=\"ds-param\"]\/span\/text()')\r\n\r\n    urllib.request.urlretrieve(single_offer_details['foto'], \"\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/image.jpeg\") #This will save photo from URL and save it locally. This will enable me to add this to my telegram\r\n\r\n    sendTelegram('Nowy Tiguan, Price: '+ single_offer_details['price']+', Details: ' + ', '.join(single_offer_details['offer_details']))\r\n\r\n    sendPhoto()\r\n\r\nreturn single_offer_details\r\n\r\ndef get_number_of_pages():\r\n\r\n#This function will just retrieve the maximum number of pages on the website. This is used when iterating through n pages\r\n\r\n    url = 'https:\/\/www.otomoto.pl\/osobowe\/volkswagen\/tiguan\/seg-suv\/od-2016\/?search%5Bfilter_float_year%3Ato%5D=2018&search%5Bfilter_float_mileage%3Ato%5D=60000&search%5Bfilter_enum_fuel_type%5D%5B0%5D=petrol&search%5Border%5D=created_at_first%3Adesc&search%5Bbrand_program_id%5D%5B0%5D=&search%5Bcountry%5D=&page=1'\r\n\r\n    headers = {'User-Agent':'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36'}\r\n\r\n    request = requests.get(url, headers = headers)\r\n\r\n    tree = html.fromstring(request.text)\r\n\r\n    max_page= tree.xpath('\/\/ul[@class=\"om-pager rel\"]\/li[last()-1]\/a\/span\/text()')[0].strip()\r\n\r\nreturn int(max_page)\r\n\r\ndef get_everything():\r\n\r\n#This function iterates through all pages, saving everything into globabl variable previous_offers that will be saves to json.\r\n\r\nfor i in range(1,get_number_of_pages()):\r\n\r\n        get_offers(i)\r\n\r\ntry:\r\n\r\nwith open('\/home\/user\/Documents\/Programowanie\/TiguanWatchOut\/tiguan.json', 'w') as json_file:\r\n\r\n            json.dump(previous_offers, json_file)\r\n\r\nexcept Exception as e:\r\n\r\n        sendTelegram(e)\r\n\r\ndef get_vin_and_phone(id):\r\n\r\n#Digging in website's code let me discover that Vin and Phone number are available under those URLs without any additional authentication\r\n\r\n    vin_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/vin\/\"\r\n\r\n    phone_url = \"https:\/\/www.otomoto.pl\/ajax\/misc\/contact\/multi_phone\/{}\/0\"\r\n\r\n    request = requests.get(vin_url+id)\r\n\r\n    vin = request.text.replace(\"\\\"\",\"\")\r\n\r\n    request = requests.get(phone_url.format(id))\r\n\r\n    phone = json.loads(request.text)[\"value\"].replace(\" \",\"\")\r\n\r\n#print(vin)\r\n\r\n#print(phone)\r\n\r\nreturn vin, phone\r\n\r\nget_everything()\r\n<\/pre>\n
 <\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Jaki\u015b czas temu zacz\u0105\u0142em rozgl\u0105da\u0107 si\u0119 za samochodem. Szybko znu\u017cy\u0142o mnie przegl\u0105danie ofert na np. otomoto.pl, wi\u0119c postanowi\u0142em zautomatyzowa\u0107 ca\u0142y proces i u\u0142atwi\u0107 sobie \u017cycie.<\/p>\n
\"\"<\/a><\/p>\n
Read More<\/a><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34],"tags":[],"class_list":["post-1382","post","type-post","status-publish","format-standard","hentry","category-python-pl"],"_links":{"self":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1382","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/comments?post=1382"}],"version-history":[{"count":5,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1382\/revisions"}],"predecessor-version":[{"id":1388,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/1382\/revisions\/1388"}],"wp:attachment":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/media?parent=1382"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/categories?post=1382"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/tags?post=1382"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}