{"id":205,"date":"2018-05-23T15:50:54","date_gmt":"2018-05-23T13:50:54","guid":{"rendered":"http:\/\/cwiok.pl\/?p=205"},"modified":"2018-05-24T07:47:13","modified_gmt":"2018-05-24T05:47:13","slug":"looping-through-a-table-to-get-data-from-web-in-powerbi","status":"publish","type":"post","link":"https:\/\/cwiok.pl\/index.php\/en\/2018\/05\/23\/looping-through-a-table-to-get-data-from-web-in-powerbi\/","title":{"rendered":"Looping through a table to get data from Web in Power BI"},"content":{"rendered":"<p>As a newly appointed Datanaut in Clouds on Mars, I work with Power BI on a daily basis. My first short assignment was to help our client with the connection to the API in Power BI. The pilot report was connected to only one web page. However, the requirement of our client was to have a report connected to multiple web pages. The list of webpages was expected to change over time (It was also downloaded from API). Tt meant iterating through the table with IDs and doing a web connection to each URL with a new ID. It was a good opportunity to use M language. There is no loop function in M, so I also had to go around this limitation.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-224\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_.jpg\" alt=\"\" width=\"1200\" height=\"628\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_.jpg 1200w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-300x157.jpg 300w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-768x402.jpg 768w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-1024x536.jpg 1024w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/p>\n<p>As an example, let us say that I want to download the weather data for multiple cities in Poland. The URL looks like this:<\/p>\n<p>https:\/\/danepubliczne.imgw.pl\/api\/data\/synop\/station\/jeleniagora<\/p>\n<p>I am interested in 7 following cities that I have already put in the table Cities:<\/p>\n<p>City<br \/>\njeleniagora<br \/>\nwarszawa<br \/>\nszczecin<br \/>\nkoszalin<br \/>\nlublin<br \/>\nkatowice<br \/>\nlegnica<\/p>\n<p>To download the data manually, I would have to import the data from a web page 7 times and merge the tables. Feasible, however would be much harder with 100 cities or a dynamic table with cities i.e. table that is also a web query. To make the process of automation clear I have broken it into several short steps:<\/p>\n<p>First step is to download the data for one of the cities and perform the transformations. This should leave us with a table in a satisfactory format. When downloading data for one city, I get this table:<br \/>\n<a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-216 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_1.png\" alt=\"\" width=\"267\" height=\"238\" \/><\/a><br \/>\nNaturally, to make the data easier to merge with the rest of the cities I have to pivot this table and rename the columns.<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-217 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2.png\" alt=\"\" width=\"1477\" height=\"56\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2.png 1477w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2-300x11.png 300w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2-768x29.png 768w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Table_2-1024x39.png 1024w\" sizes=\"auto, (max-width: 1477px) 100vw, 1477px\" \/><\/a><\/p>\n<p>Second step comes after making all of the transformations to the data. It is making a custom function that takes a city as a parameter and returns a table similar to the one above.<\/p>\n<p>To do it, I edit the M script in advanced editor:<br \/>\n<a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Advanced_editor.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-207 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Advanced_editor.png\" alt=\"\" width=\"430\" height=\"128\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Advanced_editor.png 430w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Advanced_editor-300x89.png 300w\" sizes=\"auto, (max-width: 430px) 100vw, 430px\" \/><\/a><br \/>\nThe script (below) has to have a parameter declared in the beginning and in the URL, so that it can access different webpages in each call.<br \/>\n<a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_1-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-212 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_1-1.png\" alt=\"\" width=\"1021\" height=\"134\" \/><\/a><br \/>\nBefore &#8220;let&#8221; I insert &#8220;(city as text) =&gt;&#8221;, which is a declaration of a parameter. Later, I replace &#8220;jeleniagora&#8221; with the parameter using &amp; to connect them. Finished script looks like this:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_2-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-213 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_2-1.png\" alt=\"\" width=\"943\" height=\"190\" \/><\/a><\/p>\n<p>It is very similar to the script of the initial table. I rename it to get_weather. I can call it by typing in the parameter:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_query.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-215 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_query.png\" alt=\"\" width=\"290\" height=\"178\" \/><\/a><\/p>\n<p>Third step involves calling the function repetedly for each city. I select blank query from sources and type in a script, which looks like this:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-214 aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_3.png\" alt=\"\" width=\"454\" height=\"87\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_3.png 454w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/M_3-300x57.png 300w\" sizes=\"auto, (max-width: 454px) 100vw, 454px\" \/><\/p>\n<p>City_names gives me a list of all cities I want to iterate through. By using List.Transform(city_names, each get_weather(_)) Power BI applies the function to each city from the list and merges them into a list. The result of the query should look similar to this:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-210 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_1.png\" alt=\"\" width=\"130\" height=\"177\" \/><\/a><\/p>\n<p>To get the data I transform it to table:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-211 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_2.png\" alt=\"\" width=\"352\" height=\"254\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_2.png 352w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Loop_2-300x216.png 300w\" sizes=\"auto, (max-width: 352px) 100vw, 352px\" \/><\/a><\/p>\n<p>And expand:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Expand_1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-208 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Expand_1.png\" alt=\"\" width=\"369\" height=\"439\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Expand_1.png 369w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Expand_1-252x300.png 252w\" sizes=\"auto, (max-width: 369px) 100vw, 369px\" \/><\/a><\/p>\n<p>This gives me a table with the data for all the cities:<\/p>\n<p><a href=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-209 size-full aligncenter\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final.png\" alt=\"\" width=\"1485\" height=\"198\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final.png 1485w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final-300x40.png 300w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final-768x102.png 768w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/Final-1024x137.png 1024w\" sizes=\"auto, (max-width: 1485px) 100vw, 1485px\" \/><\/a><\/p>\n<p>This approach can be used with other connectors as well. Important thing to note is the ability to iterate through a list with List.Transform() function. If you know any other way to perform this action, let me know!<\/p>\n<p>Thanks<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today&#8217;s post shows how to call a custom function for each row in a table. This can be done both using standard Power BI interface and M language scripts. I have decided to try the latter.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-224\" src=\"http:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_.jpg\" alt=\"\" width=\"1200\" height=\"628\" srcset=\"https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_.jpg 1200w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-300x157.jpg 300w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-768x402.jpg 768w, https:\/\/cwiok.pl\/wp-content\/uploads\/2018\/05\/artyku\u0142_01_-1024x536.jpg 1024w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/p>\n<div class=\"tech_read_more\"><a href=\"https:\/\/cwiok.pl\/index.php\/en\/2018\/05\/23\/looping-through-a-table-to-get-data-from-web-in-powerbi\/\">Read More<\/a><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26],"tags":[],"class_list":["post-205","post","type-post","status-publish","format-standard","hentry","category-powerbi"],"_links":{"self":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/205","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/comments?post=205"}],"version-history":[{"count":0,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/205\/revisions"}],"wp:attachment":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/media?parent=205"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/categories?post=205"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/tags?post=205"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}