Key vault secrets in ADF pipelines

2021-03-18

4 min

In this article

This short post looks at some considerations when using key vault secrets in Data Factory to securely pass information in pipeline activities. This is not an exhaustive list however but do take note.

Sensitive pipeline information

There will be times where sensitive information needs to be passed to various activities within pipelines and although Data Factory does offer a solution for this, it is a little awkward to use.

The solution to use key vault secrets within pipeline activities is great for a variety of reasons as noted in the following list:

Benefits

Ability to hide sensitive information.
Minimises the negative impact on deployment complexity.
Can be integrated in dynamic linked service connections, allowing for code re-use.

Calling secrets in pipelines

Set up

Microsoft have an example of calling secrets in pipelines and it is simple enough to follow so head over there to see how to set this up:

Use Azure Key Vault secrets in pipeline activities - Azure Data Factory | Microsoft Docs

Additional activity

In addition to the Microsoft example above, I have added a secret in key vault, a linked service to an Azure SQL database, associated dataset and a look up pipeline activity with the following dynamic content @concat(‘SELECT’, ‘’’’, activity(‘get-pipeline-secret’).output.value, ‘’’’).

This addition is for a comparison with the “set variable” activity we created earlier from the Microsoft Docs link, so do set this up to follow along.

Considerations

To summarise, these are some of the things to consider when passing secrets in ADF pipelines:

1. Failing to specify API version.

2. Plain text logging

3. Activities that support secure input and output.

Failing to specify API version

You must add: “?api-version=7.0” to the end of your secret URI or an error like the below

will be returned as shown in the following image:

Specify API version error

Plain text logging

Check the “Secure Output” option to prevent plain text logging of secret values.

If this is not enabled, all the work done to secure passwords in key vault and pass those secrets securely in pipeline activities will, arguably, be in vain.

Without secure output

Issue a debug run of the pipeline created earlier and set the web activity created earlier in the set-up section (see above) to have no secure output. Observe, as the plain text value of the secret value is visible in the output of the “get-pipeline-secret” web activity log, under the value section. Howzat! for security.

Without secure input or output

With secure input and output

Check the “Secure Input” and “Secure Output” options for the web activity (get-pipeline secret in my case) to plain text logging of secret values in and hey presto! No more plain text logging of secrets.

With secure input or output

Oh, Dear!

Except, that is not the case for dependent activities. Unfortunately, the key vault secrets are still in plain text for dependent activities even though the initial web activity had the secure input and output options turned on.

This is shown in the following screenshots:

Dependent activities plain text logging

With secure input and output

As in the previous section, check the “Secure Input” and “Secure Output” options for the web activity and additionally, do this for dependent activities.

If you hit a snag with the set variable activity, it is because not all ADF activities have this option by default. If you are to call key vault secrets in pipeline activities, be really careful with this.

The screenshots below show this reality for the lookup and set variable activities we have:

Secure logging and plain text logging

Final thoughts

Additional care must be taken when using key vault secrets in pipelines, not just for setting the secure input and output options but for making sure the activities to use these secrets support the secure options.

Although slightly awkward to set up, the process to call ADF secrets from within pipelines is great to have than not and should provide some comfort that sensitive information is being treated with some care.

Resources