Create Azure Logic Apps Custom Connector For Azure Databricks

This article guides you on how to create logic apps custom connector for Azure Databricks. Azure Logic Apps is a cloud service that helps you schedule, automate, and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organisations. There are few services similar to azure logic apps like Microsoft Power Automate, and Microsoft Power Apps.

Azure logic apps provides number of in built connectors , however we can design and create our own connector for integrating different services by using Azure’s Logic Apps Custom Connector service. This article shows how you can build a custom connector that can be used to trigger a job on Azure Databricks.

To create a custom connector, you must describe the API you want to connect to so that the connector understands the API’s operations and data structures. A custom connector is a wrapper around a REST API (Logic Apps also supports SOAP APIs) that allows Logic Apps to communicate with that REST or SOAP API. Choose API end point as REST and Open API file , have kept all with default values in this section.

6. Next section is General information, in this you can add relevant icon for your connector, here i have used Databricks icon, you can optionally change the colour and add description for your connector. Next 3 options are scheme, host and base URL. These 3 options are used to create URL of your service so here we have to provide values based on Azure Databricks URL.

Azure Databricks has a URL of form https://<databricksinstancename>.azuredatabricks.net based on this

scheme: https

Host :<databricksinstancename>.azuredatabricks.net

Base URL you can keep as / or you can use /api/2.0(as we are using Databricks api , we are using this endpoint) , this part can be alternatively done in POST request in further sections.

7. Next section is Authentication Type here we can provide different authentication methods like Basic, OAuth 2 and Api Key. For Databricks we are to use basic authentication method by using personal access token authentication. Also you have to give labels for your username and password i have kept as default. Please note that we are not using any credentials here. These are just display labels for authentication.

8. Next section is Definition, here we will define action for triggering the job on Azure Databricks, you can add multiple actions and triggers in this section.However we are going to create only one action for this article purpose.

a) Click on New Action in the left panel , it will prompt a new wizard. Fill the relevant Summary and Description here. operation Id.

Leave the Visibility property set to none. This property for operations and parameters in a logic app or flow has the following options:

After this click on import.

C) Next section is Validation , this section helps you identify potential issues with this action. If all fields are correctly configured it shows validation succeed else it shows error.

9.After this you have to save the connector details by using Update Connector option at top right corner. This step will create the connector ready to be used in Azure Logic Apps

10. To Use this Connector , Go to azure logic apps instance, go to logic app designer , first select one trigger of type Schedule and add it to logic app.

11. After this click on new step and select custom connector option , here your Databricks Connector should be visible under action tab that you have successfully created.

12. Select the custom connector , provide relevant connection name, As we are using Databricks personal access token method of authentication username is ‘Token’ and password is token that you have to create in Databricks.

How to create personal access token in Azure Databricks?

A)Go to Azure Databricks service and launch your workspace.

B)Click on user icon at top right corner and choose user settings from drop-down list.

C)Click on Generate New Token and fill the comment and lifetime for token. After this click on Generate, this will show Token Value Only once and you have to Copy this as you wont be able to see it again.After this click on Done.Your token will be now listed with Token ID, comment , Creation and Expiration date. Note that Token ID is just unique identifier to your token and not the key or password. The key or password is one which you have copy pasted while token was getting generated.

13. After this click on create , this will open another wizard to fill he inputs related to you job based on the structure of request body which was made during creation of connector. Here i am going trigger the job having job ID 5.

14. After this click on save. This will trigger the Databricks job based on schedule or you can alternatively click on Run option to test the logic app immediately .

This will trigger the job in Databricks.

Related posts:

pernah di suatu waktu, si empunya diri ini terdesak. Didesak oleh pilihan yang ia ambil sendiri, oleh amanah amanah yang dia pikir diawal, “udah jalanin dulu aja gengs" .. dan ternyata kini datang…

Bulan Juli adalah bulan yang CERIA bagi anak-anak Matalangi. Tepatnya pada tanggal 12 Juli 2018 anak-anak Matalangi mengunjungi Kampung Main Cipulir. Anak-anak yang berkesempatan ikut dalam…

This is obviously an extremely uncomfortable topic for many and it is equally tricky to write about however the need for us to be having this conversation far out ways any feelings of it being a…