Due to the demand I have decided to revisit the solution that deletes files from FTP from within a Data Factory pipeline. I believe this approach is a much cleaner design. It is also easier to debug and control.
Key Vault is an Azure module, which lets you safely store your secrets/keys. This way your developers do not have to store credentials or connection strings in the code, but can simpy ask Key Vault to provide them. The application will be handed the password and will be able to log in without the developer even knowing it. Let us create a key vault!
Cognitive Services from Microsoft offer pretrained networks that allow citizen developers to quickly develop e.g. image recognition algorithms. This short post is to show how it is done and how I have created a real-time people recognition system. The goal of this solution is to recognize people in our office and possibly inform our beloved founders of this. What I used was: a python script and a FullHD webcam + Cognitive Services Custom Vision API.
This tutorial was requested by one of our clients. His app was connected to DWH, which was the source for the Analysis Services. After the data was input into the edit form, the task was to start processing the table to see it through Power BI.
During development of an ETL process in Azure Data Factory, you may come across a need to delete the data from FTP. It may be neccessary e.g. when you want to make place for new incoming data or you don’t want your pipeline to pull the old data twice. The solution of the problem lays in a custom activity.
