{"id":108,"date":"2017-07-30T20:08:55","date_gmt":"2017-07-30T18:08:55","guid":{"rendered":"http:\/\/cwiok.pl\/?p=108"},"modified":"2018-10-22T11:24:02","modified_gmt":"2018-10-22T09:24:02","slug":"usuwanie-plikow-z-ftp-uzywajac-azure-data-factory","status":"publish","type":"post","link":"https:\/\/cwiok.pl\/index.php\/pl\/2017\/07\/30\/usuwanie-plikow-z-ftp-uzywajac-azure-data-factory\/","title":{"rendered":"Usuwanie plik\u00f3w z FTP u\u017cywaj\u0105c Azure Data Factory"},"content":{"rendered":"<p>Podczas budowania procesu ETL w Azure Data Factory mo\u017cemy napotka\u0107 konieczno\u015b\u0107 wyczyszczenia pobranych danych z serwera FTP. B\u0119dzie to wymagane np. kiedy nale\u017cy zrobi\u0107 miejsce dla nowych danych lub aby pipeline nie za\u0142adowa\u0142 starych danych. Rozwi\u0105zaniem tego problemu jest niestandardowa aktywno\u015b\u0107 w Data Factory.<\/p>\n<h1><!--more--><br \/>\nZauwa\u017cy\u0142em, \u017ce coraz wi\u0119cej os\u00f3b u\u017cywa tego poradnika, wi\u0119c stworzy\u0142em now\u0105 od\u015bwie\u017con\u0105 wersj\u0119. Mo\u017cesz j\u0105 znale\u017a\u0107 <a href=\"http:\/\/cwiok.pl\/index.php\/pl\/2018\/10\/22\/usuwanie-plikow-z-ftp-przy-pomocy-azure-data-factory-podejscie-drugie\/\">tutaj.<\/a><\/h1>\n<p>Stworzenie niestandardowej aplikacji jest opisane w poradniku napisanym przez Microsoft dost\u0119pnym <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/data-factory\/data-factory-use-custom-activities\">tutaj<\/a>.<\/p>\n<p>By mie\u0107 mo\u017cliwo\u015b\u0107 usuwania plik\u00f3w z FTP lub czyszczenia danego folderu w C# nale\u017cy zastosowa\u0107 kod:<\/p>\n<pre>using System.IO;\r\nusing System.Collections.Generic;\r\nusing Microsoft.Azure.Management.DataFactories.Models;\r\nusing Microsoft.Azure.Management.DataFactories.Runtime;\r\nusing System.Net;\r\nusing System.Linq;\r\n\r\nnamespace DeleteFromFTP\r\n{\r\n    public class DeleteFromFTP : IDotNetActivity\r\n    {\r\n        public IDictionary&lt;string, string&gt; Execute(\r\n        IEnumerable linkedServices,\r\n        IEnumerable datasets,\r\n        Activity activity,\r\n        IActivityLogger logger)\r\n        {\r\n\t\t\t\/\/Creating refrences for the pipeline\r\n            DotNetActivity dotNetActivity = (DotNetActivity)activity.TypeProperties;\r\n            string Path = dotNetActivity.ExtendedProperties[\"Path\"];\r\n            string IP = dotNetActivity.ExtendedProperties[\"IP\"];\r\n            string Login = dotNetActivity.ExtendedProperties[\"Login\"];\r\n            string Password = dotNetActivity.ExtendedProperties[\"Password\"];\r\n            AzureStorageLinkedService linkedService = linkedServices.First(ls =&gt; ls.Name == dotNetActivity.PackageLinkedService).Properties.TypeProperties as AzureStorageLinkedService;\r\n\r\n            DeleteFTPDirectory(Path+\"\/\", IP, Login, Password);\r\n\r\n            \r\n            return new Dictionary&lt;string, string&gt;();\r\n        }\r\n        public static List DirectoryListing(string Path, string ServerAdress, string Login, string Password)\r\n        {\r\n            FtpWebRequest request = (FtpWebRequest)WebRequest.Create(\"ftp:\/\/\" + ServerAdress + Path);\r\n            request.Credentials = new NetworkCredential(Login, Password);\r\n\r\n            request.Method = WebRequestMethods.Ftp.ListDirectory;\r\n\r\n            FtpWebResponse response = (FtpWebResponse)request.GetResponse();\r\n            Stream responseStream = response.GetResponseStream();\r\n            StreamReader reader = new StreamReader(responseStream);\r\n\r\n            List result = new List();\r\n\r\n            while (!reader.EndOfStream)\r\n            {\r\n                result.Add(reader.ReadLine());\r\n            }\r\n\r\n            reader.Close();\r\n            response.Close();\r\n\r\n            return result;\r\n        }\r\n        public static void DeleteFTPFile(string Path, string ServerAdress, string Login, string Password)\r\n        {\r\n            FtpWebRequest clsRequest = (System.Net.FtpWebRequest)WebRequest.Create(\"ftp:\/\/\" + ServerAdress + Path);\r\n            clsRequest.Credentials = new System.Net.NetworkCredential(Login, Password);\r\n\r\n            clsRequest.Method = WebRequestMethods.Ftp.DeleteFile;\r\n\r\n            string result = string.Empty;\r\n            FtpWebResponse response = (FtpWebResponse)clsRequest.GetResponse();\r\n            long size = response.ContentLength;\r\n            Stream datastream = response.GetResponseStream();\r\n            StreamReader sr = new StreamReader(datastream);\r\n            result = sr.ReadToEnd();\r\n            sr.Close();\r\n            datastream.Close();\r\n            response.Close();\r\n        }\r\n        public static void DeleteFTPDirectory(string Path, string ServerAdress, string Login, string Password)\r\n        {\r\n            FtpWebRequest clsRequest = (System.Net.FtpWebRequest)WebRequest.Create(\"ftp:\/\/\" + ServerAdress + Path);\r\n            clsRequest.Credentials = new System.Net.NetworkCredential(Login, Password);\r\n\r\n            List filesList = DirectoryListing(Path, ServerAdress, Login, Password);\r\n\r\n            foreach (string file in filesList)\r\n            {\r\n                DeleteFTPFile(Path + file, ServerAdress, Login, Password);\r\n            }\r\n\r\n\r\n        }\r\n    }\r\n}<\/pre>\n<p>Nast\u0119pnie po skompilowaniu kodu aktywno\u015b\u0107 w Data Factory powinna mie\u0107 tak\u0105 posta\u0107:<\/p>\n<pre>{\r\n    \"name\": \"Delete_from_FTP\",\r\n    \"properties\": {\r\n        \"description\": \"Activity to delete files from a folder\",\r\n        \"activities\": [\r\n            {\r\n                \"type\": \"DotNetActivity\",\r\n                \"typeProperties\": {\r\n                    \"assemblyName\": \"DeleteFromFTP.dll\",\r\n                    \"entryPoint\": \"DeleteFromFTP.DeleteFromFTP\",\r\n                    \"packageLinkedService\": ,\r\n                    \"packageFile\": ,\r\n                    \"extendedProperties\": {\r\n                        \"Path\": ,\r\n\t\t\t\t\t\t\"IP\": ,\r\n\t\t\t\t\t\t\"Login\": ,\r\n\t\t\t\t\t\t\"Password\": ,\r\n                    }\r\n                },\r\n                \"inputs\": [\r\n                    {\r\n                        \"name\": \"Input Dataset\r\n                    }\r\n                ],\r\n                \"outputs\": [\r\n                    {\r\n                        \"name\": \"Output Dataset\"\r\n                    }\r\n                ],\r\n                \"policy\": {\r\n                    \"timeout\": \"01:00:00\",\r\n                    \"concurrency\": 1,\r\n                    \"longRetry\": 10,\r\n                    \"longRetryInterval\": \"01:00:00\"\r\n                },\r\n                \"scheduler\": {\r\n                    \"frequency\": \"Day\",\r\n                    \"interval\": 1\r\n                },\r\n                \"name\": \"Delete_from_FTP\",\r\n                \"linkedServiceName\": \"AzureBatchLinkedService\"\r\n            }\r\n        ],\r\n        \r\n        \"isPaused\": false,\r\n        \"hubName\": \"Your_hub_name\",\r\n        \"pipelineMode\": \"Scheduled\"\r\n    }\r\n}\r\n<\/pre>\n<p>Je\u015bli nie chcecie podawa\u0107 loginu i has\u0142a w aktywno\u015bci, wtedy nale\u017cy zrobi\u0107 to w ju\u017c w kodzie C#.<\/p>\n<p>Kod prosty i przetestowany \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Podczas budowania procesu ETL w Azure Data Factory mo\u017cemy napotka\u0107 konieczno\u015b\u0107 wyczyszczenia pobranych danych z serwera FTP. B\u0119dzie to wymagane np. kiedy nale\u017cy zrobi\u0107 miejsce dla nowych danych lub aby pipeline nie za\u0142adowa\u0142 starych danych. Rozwi\u0105zaniem tego problemu jest niestandardowa aktywno\u015b\u0107 w Data Factory.<\/p>\n<div class=\"tech_read_more\"><a href=\"https:\/\/cwiok.pl\/index.php\/pl\/2017\/07\/30\/usuwanie-plikow-z-ftp-uzywajac-azure-data-factory\/\">Read More<\/a><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-108","post","type-post","status-publish","format-standard","hentry","category-azure-pl"],"_links":{"self":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/comments?post=108"}],"version-history":[{"count":0,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/posts\/108\/revisions"}],"wp:attachment":[{"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/media?parent=108"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/categories?post=108"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cwiok.pl\/index.php\/wp-json\/wp\/v2\/tags?post=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}