# Today I Learned This is the site where I share little snippets and tidbits of learnings that don't deserve a full blog post on https\://cazzulino.com, grouped by areas, rather than by date. # How to get messages logged in dotnet build output \ gone? I learned that the reason my high-importance messages were totally gone from the `dotnet build` output is due to the new [terminal logger](https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/9.0/terminal-logger) being the new default. To bring the old logger (and those messages back), you need to pass `-tl:off` now, or set `MSBUILDTERMINALLOGGER=false` [envvar](https://github.com/dotnet/msbuild/blob/main/documentation/terminallogger/Opt-In-Mechanism.md). You can set this for all your local builds in a folder by creating an `Directory.Build.rsp` with the `-tl:off` switch which is [picked up automatically by MSBuild](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-response-files?view=visualstudio). # How to emit descriptions for exported JSON schema using JsonSchemaExporter We now (.NET 9+) have an API to export a JSON schema from a .NET type:
The default exporter will not, however, provide descriptions for those properties, even if a `[Description(...)]`attribute is provided. The way to fix that is to provide a `TransformSchemaNode` callback like so: ```csharp var node = JsonSchemaExporter.GetJsonSchemaAsNode(options, typeof(Product), new JsonSchemaExporterOptions { TreatNullObliviousAsNonNullable = true, TransformSchemaNode = (context, node) => { var description = context.PropertyInfo?.AttributeProvider?.GetCustomAttributes(typeof(DescriptionAttribute), false) .OfType() .FirstOrDefault()?.Description; if (description != null) node["description"] = description; return node; }, }); ``` # NuGet NuGet-related learnings # Suppress dependencies when packing Some packing scenarios, especially those involving tools, require ignoring all dependencies (PackageReference as well as framework references) from the resulting package dependencies. I [learned of a property](https://github.com/NuGet/Home/issues/6354) supported by `dotnet pack` for this: ```markup true ``` NuGetizer has more flexibility in this regard, providing both a `$(PackFrameworkReferences)` property as well as granular control over referenced packages via the `Pack` metadata on each `PackageReference`: ```markup false false ``` But in order to simplify this scenario further, both the compatibility `SuppressDependenciesWhenPacking` property as well as a new `PackDependencies` property is supported in [nugetizer](https://www.nuget.org/packages/nugetizer) to achieve the same. # Hide contentFiles from your nuget packages When you package compile items with your package (i.e. `.cs)`, they are visible by default in your consumer's project. That's not always what you want. From [a github issue on NuGet](https://github.com/NuGet/Home/issues/4856#issuecomment-288151716), I learned a neat trick to hide all the files in your package, automatically! All you need is to include a .props file in your package `build` folder (i.e. if your package is named `Foo`, make sure the file ends up `build\Foo.props`) ```markup false ``` This works because NuGet generates a .props file containing all your contentFiles items, under the `obj` folder, which contains both pieces of metadata used above to filter the visibility on items coming from our package. # Packaging transitive analyzers with NuGet Consider the scenario of [ThisAssembly](https://github.com/kzu/ThisAssembly) and its referenced packages: the main package is essentially a meta-package so that anyone wanting to leverage all the codegen in all the ThisAssembly.\* packages can reference a single one. By default, NuGet pack will create a package that declares the project reference dependencies like so: ```markup ``` Note all those `exclude`. Not good since now it means projects referencing this package will not get the transitive analyzers, which are precisely the point of this meta-package. The fix is [highly non-obvious](https://github.com/NuGet/Home/issues/3697#issuecomment-342983009): you must explicitly state that *none* of the referenced project assets are to be flagged as private: ```markup ``` This properly allows the transitive build and analyzer assets to be properly installed on the referencing project. # How to add search to static nuget feed I have been using [static serverless nuget](https://www.cazzulino.com/serverless-nuget-feed.html) feeds for a while now. [Sleet](https://github.com/emgarten/Sleet) is totally awesome. One missing piece was search: searching a static feed would just return everything, always. [Not anymore](https://github.com/emgarten/Sleet/pull/142)! To set it up: 1. Fork [Sleet.Search](https://github.com/emgarten/Sleet.Search): this is the Azure function project that will perform the search on your static feeds. You'll host your own in a consumption (aka serverless) plan in Azure. 2. Create an Azure Functions app and follow the [deployment steps from my blog](https://www.cazzulino.com/minimalist-shortlinks.html#deployment): that's basically setting up the simplest CI/CD for a GH repo. 3. By default, the project allows passing in the static feed "search" url (which is a blob named under `/search/query` in your sleet blob container) and requires function authentication. In my case, I wanted a simpler search URL and anonymous access, so I [made that change to Sleet.Search](https://github.com/kzu/Sleet.Search/commit/99c736b). Basically the function can now be accessed at `/query`, without URL-encoding another URL parameter there. 4. Follow instructions to [enable search for your sleet feed via settings](https://github.com/emgarten/Sleet/blob/master/doc/external-search.md). That's it. Now you can configure my feed and it will be fully searchable. Plus, I used [Azure Functions Proxies](https://docs.microsoft.com/en-us/azure/azure-functions/functions-proxies) to make the blob storage URL that much nicer. I even made the query URL nice, since it was trivial too: 😍. # Populate RepositoryBranch in CI for NuGet Pack I learned that [RepositoryBranch](https://github.com/dotnet/sourcelink/issues/188#issuecomment-427975234) is the only NuGet Pack-related property not populated already by the .NET SDK+SourceLink. So instead of going for a full-blown (and typically over-blown) solution for build/assembly versioning from Git information (such as GitInfo or GitVersion or the myriad others), you can trivially pass in this value from your CI script with: ``` dotnet pack -p:RepositoryBranch=${GITHUB_REF#refs/*/} ``` This works in all OSes if you're running with a bash shell. On a Windows agent, you'd need to opt-in to that using `shell: bash` on the `run` action (if you're using GitHub Actions). Or you can also just [default to bash for all run actions](https://github.com/kzu/oss/blob/main/.github/workflows/build.yml#L14-L16). NOTE: the wildcard instead of `heads` is so that this also works for tags, in which case the "branch" will be the tag name. # Ignore folder from dotnet-format The easiest way to ignore an entire folder (i.e. one containing files you're syncing from an external repository such as [catbag](https://github.com/devlooped/catbag) using [dotnet-file](https://www.nuget.org/packages/dotnet-file)) is to create a `.editorconfig` file with the following content: ```editorconfig [*] generated_code = true ``` # Accessing Tor .onion URLs via HttpClient with .NET6 In .NET6, there is built-in [support for SOCKS4/5 proxies](https://github.com/dotnet/runtime/pull/48883)! This means you can install the [Tor](https://dist.torproject.org/torbrowser/) service and without anything else, access any `.onion` URL with plain `HttpClient`: ```csharp using System; using System.Net; using System.Net.Http; var http = new HttpClient(new HttpClientHandler { Proxy = new WebProxy("socks5://127.0.0.1:1338") }); Console.WriteLine(await http.GetStringAsync("https://bbcnewsv2vjtpsuy.onion")); ``` NOTE: I could not get this to work with gRPC, which requires HTTP/2. You can get the daily .NET6 SDK installer permalinks from: | Platform | Main | | ------------------- | :--------------------------------------------------------: | | **Windows (x64)** | | | **Windows (x86)** | | | **Windows (arm64)** | | | **macOS (x64)** | | | **macOS (arm64)** | | # Installing .NET 5.0 on Raspberry Pi 4 None of the official methods worked for me (not even [involving horrid fixes](https://github.com/dotnet/core/issues/4446#issuecomment-843162084) ;)). So I ended up going the super manual route (mostly copying and ). ``` $ sudo apt-get install lshw $ sudo lshw description: Computer product: Raspberry Pi 4 Model B Rev 1.4 serial: 10000000d5e618a2 width: 64 bits ; OK, it's 64bits ; Get Arm64 download URL from https://dotnet.microsoft.com/download/dotnet/5.0 $ mkdir temp && cd temp $ curl [URL] --output [FILENAME] $ mkdir $HOME/dotnet $ sudo tar zxf [FILENAME] -C $HOME/dotnet/ $ sudo ln -s $HOME/dotnet/dotnet /usr/local/bin $ dotnet --version 5.0.203 ; or whatever version you downloaded ; Now we need to add it to PATH. Add the exports $ sudo nano ~/.bashrc export DOTNET_ROOT=$HOME/dotnet export PATH=$PATH:$HOME/dotnet export PATH=$PATH:$HOME/.dotnet/tools ; then Ctrl+X to save and exit. finally, load the new exports $ source ~/.bashrc ``` # Quickly check C# compiler and language version Just add `#error version` anywhere in a C# file and you'll see a tooltip with the information: ![](/files/aF5u8AIb3GNQa79qpG9h) # Disable diagnostic analyzers for entire folder/submodules When you have submodules in your repository, it's not uncommon for those to be subject to a different quality bar than your main project. You may not even have a say in fixing issues upstream, or conventions may be totally different even. In this case, you can introduce an `.editorconfig` in your repository which turns off all analyzer diagnostics for an entire directory path (and all subdirectories) by leveraging wildcards. Say your submodules are in a folder `ext` a couple levels up from your `.editorconfig`: ``` [../../ext/**/*.cs] dotnet_analyzer_diagnostic.severity = none ``` Learned from [this commit](https://github.com/AArnott/Nerdbank.Streams/commit/50e322a90903283191ccaf67a8f27cf5aba6784c) after following it from a related [roslyn issue](https://github.com/dotnet/roslyn/issues/42762). # Persisting output files from source generators Some time back, I had my own I/O code that [based on some MSBuild property would persist my generated source](https://github.com/kzu/ThisAssembly/commit/65d43b2f6) for troubleshooting. This is no longer needed since you can now set ```markup true ``` That will emit the sources to the `$(IntermediateOutputPath)/generated/[GeneratorAssembly]/[GeneratorTypeFullName]` folder by default. If you want to also change where the generated sources are placed, you can additionally set the `CompilerGeneratedFilesOutputPath` property. # Use C# 9 records in non-net5.0 projects The new C# 9 records syntax is quite nice: ```csharp [DebuggerDisplay("{Name} = {Value}")] record ResourceValue(string Name, string Value, bool HasFormat) { public bool IsIndexed { get; init; } public List Format { get; } = new List(); } ``` When using it in a non-*net5.0* project (i.e. *netstandard2.0*), [compilation fails with an error](https://github.com/dotnet/roslyn/issues/45510): ``` Predefined type 'System.Runtime.CompilerServices.IsExternalInit' is not defined or imported ``` To workaround the issue, simply declare the missing type in your project: ```csharp // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. // See the LICENSE file in the project root for more information. using System.ComponentModel; namespace System.Runtime.CompilerServices { /// /// Reserved to be used by the compiler for tracking metadata. /// This class should not be used by developers in source code. /// [EditorBrowsable(EditorBrowsableState.Never)] sealed class IsExternalInit { } } ``` ([as seen elsewhere too](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Text.Json/tests/Serialization/IsExternalInit.cs)). Note the class doesn't even need to be public. If you're multitargeting net5.0 and other TFMs, just add this bit of MSBuild to remove it for net5.0 since it's built-in: ```markup ``` # AsyncLocal never leaks and is safe for CallContext-like state Even if it's typically used in a static field, the values never leak since they are bound to a transient ExecutionContext Sometime ago I wrote on [How to migrate CallContext to .NETStandard and .NETCore](https://www.cazzulino.com/callcontext-netstandard-netcore.html), and one question mentioned that the values themselves might leak, which actually is not the case, as shown here, due to the "magic" that is AsyncLocal in combination with the [transient nature of the ExecutionContext](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs#L133-L200): ``` static AsyncLocal local = new AsyncLocal(); async Task Main() { WeakReference data = null; await Task.Run(() => { var o = new object(); data = new WeakReference(o); // We assign to the static async local, to see if it leaks. local.Value = o; }); // After execution is finished, we do have a live reference still. System.Diagnostics.Debug.Assert(data.IsAlive == true); GC.Collect(); // But a GC proves nobody is holding a strong reference to it. System.Diagnostics.Debug.Assert(data.IsAlive == false); } ``` # Using HashCode in .NETFramework How to use HashCode type in full .NET, which doesn't support it So, I just learned that not even in .net472/net48, you can leverage the cool [HashCode](https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/shared/System/HashCode.cs) type from .NETCore. It **is** [mentioned in docs](https://docs.microsoft.com/en-us/dotnet/api/system.hashcode?view=dotnet-plat-ext-3.1) as available in ".NET Platform Extensions 3.1", which isn't quite clear what it means. After a quick search here and there, I found out the [Microsoft.Bcl.HashCode](https://www.nuget.org/packages/Microsoft.Bcl.HashCode/) package, which seems to be one such ".NET platform extension" (although the version # doesn't match either :/). With that, I can now make a `KeyValuePairComparer`to check for dictionaries equality like: ``` public class KeyValuePairComparer : IEqualityComparer> { public static KeyValuePairComparer Default { get;} = new KeyValuePairComparer(); public bool Equals(KeyValuePair x, KeyValuePair y) => object.Equals(x.Key, y.Key) && object.Equals(x.Value, y.Value); public int GetHashCode(KeyValuePair obj) => HashCode.Combine(obj.Key, obj.Value); } ``` To be used as: ``` IReadOnlyDictionary first = ... IReadOnlyDictionary second = ... return first.SequenceEquals(second, KeyValuePairComparer.Default); ``` # How to locate dotnet How to locate dotnet from the currently running .NET Core application I've seen a bunch of `DotNetMuxer` implementations, but they all look quite similar (if not exactly the same). * [ASP.NET Core](https://github.com/dotnet/aspnetcore/blob/master/src/Shared/CommandLineUtils/Utilities/DotNetMuxer.cs) (same used by others, like [Orleans](https://github.com/dotnet/orleans/blob/master/src/Orleans.CodeGenerator.MSBuild.Tasks/DotNetMuxer.cs)) * [.NET runtime](https://github.com/dotnet/runtime/blob/master/src/libraries/Common/src/Extensions/CommandLineUtils/Utilities/DotNetMuxer.cs), [.NET extensions](https://github.com/dotnet/extensions/blob/master/src/Shared/src/CommandLineUtils/Utilities/DotNetMuxer.cs) * [Azure WebJobs SDK](https://github.com/Azure/azure-webjobs-sdk/blob/master/src/Analyzer/ExtensionViewer/DotNetMuxer.cs) (same as [Azure Functions SDK](https://github.com/Azure/azure-functions-vs-build-sdk/blob/master/src/Microsoft.NET.Sdk.Functions.MSBuild/Tasks/DotNetMuxer.cs)) * [.NET command-line-api](https://github.com/dotnet/command-line-api/blob/master/src/System.CommandLine.Suggest/DotnetMuxer.cs) Seems like there are two approaches: those that try to locate a `FX_DEPS_FILE` file and those that just use the current process' `MainModule` (which would be `dotnet[.exe]` itself for a .NET Core app. The latter looks simpler and the former seems unnecessary, until you take into account [self-contained .NET Core apps](https://docs.microsoft.com/en-us/dotnet/core/deploying/#publish-self-contained) or even [trimmed self-contained](https://docs.microsoft.com/en-us/dotnet/core/deploying/trim-self-contained), but even that version doesn't work in those cases :( # Conditional unit tests How to run xunit tests conditionally depending on the environment that is running the tests Turns out that there aren't any conditional `FactAttribute` in [xunit](https://github.com/xunit/xunit), but you can pretty much copy wholesale the extensions created by the [ASP.NET Core team from their repo](https://github.com/dotnet/runtime/tree/master/src/libraries/Common/tests/Extensions/TestingUtils/Microsoft.AspNetCore.Testing/src/xunit). It allows you to do things like: ``` [SkipOnCI] [ConditionalFact] public async Task SomeTest() ``` to avoid running a test entirely if you're on CI. the implementation didn't take into account CI builds running via GH actions/workflows, so I tweaked the [OnCI() method](https://github.com/dotnet/runtime/blob/master/src/libraries/Common/tests/Extensions/TestingUtils/Microsoft.AspNetCore.Testing/src/xunit/SkipOnCIAttribute.cs#L38) like so to also consider the commonly used `CI` envvar too: ``` public static bool OnCI() => (bool.TryParse(Environment.GetEnvironmentVariable("CI"), out var ci) && ci) || OnHelix() || OnAzdo(); ``` I didn't think it was necessary to create an `OnGitHub` method since the envvar is just `CI` and well, the method is already called `OnCI` :) Another alternative is using the [Xunit.SkippableFact nuget package](https://github.com/AArnott/Xunit.SkippableFact). # Skip tagged scenarios in SpecFlow with Xunit How to skip scenarios with a certain tag from executing First apply a tag of your choosing to flag scenarios that are still not ready for running: ``` Feature: Skipping scenarios by tag @Draft Scenario: This is not ready yet Given Something not done yet Then It should not fail the CI build (yet) ``` Now in the `[Binding]` class for any of the steps in the scenario, create a constructor that receives the `ScenarioContext` as follows: ``` public Steps(ScenarioContext context) { Skip.If(context.ScenarioInfo.Tags.Contains("Draft")); } ``` That's it. Turns out that the generated test for a scenario is annotated with `[SkippableFact]` so you can just skip from anywhere during the test run. From [SkippableFact](https://www.nuget.org/packages/Xunit.SkippableFact/) package. # How to get user home dir \~ cross-platform It would be great if you could just use `$(~)`, say, but that would be too good to be true :). Here's how you can get the user profile directory in both Unix-like OS (Linux/Mac) and Windows: ```markup $(HOME) $(USERPROFILE) ``` # Modifying the build for every solution in a repository Just like you can have [Directory.Build.props and Directory.Build.targets](https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build?view=vs-2019#directorybuildprops-and-directorybuildtargets) to customize your projects' build, you can also use `Directory.Solution.props` and `Directory.Solution.targets` to customize your solutions (command-line) builds. Just like the original (older?) mecanisms, [Visual Studio will not load those customizations either](https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build?view=vs-2019#customize-the-solution-build), however. In order to inspect how and where they are included in the build, it's useful to set use the [troubleshooting technique](https://docs.microsoft.com/en-us/visualstudio/msbuild/how-to-build-specific-targets-in-solutions-by-using-msbuild-exe?view=vs-2019#troubleshooting) of setting the envvar `MSBUILDEMITSOLUTION=1` and run a build. You can inspect the *.metaproj* MSBuild project generated from the solution, where you will see the imported projects. For a Directory.Solution.props with: ```markup from-solution.props ``` And a Directory.Solution.targets with: ```markup from-solution.targets ``` You will see a *.metaproj* similar to: ```markup C:\Program Files\dotnet\sdk\5.0.100-rc.2.20479.15\Roslyn <_DirectorySolutionPropsFile>Directory.Solution.props <_DirectorySolutionPropsBasePath>C:\Code\kzu\moq C:\Code\kzu\moq\Directory.Solution.props from-solution.props Debug Any CPU ... <_DirectorySolutionTargetsFile>Directory.Solution.targets <_DirectorySolutionTargetsBasePath>C:\Code\kzu\moq C:\Code\kzu\moq\Directory.Solution.targets from-solution.targets ... ... ``` Notice how the targets aren't imported but rather embedded in specific places (top of property group for .props-declared properties, bottom of property group for .targets-declared properties, and before Build target for targets). If you have properties, they are actually even evaluated before being embedded in the file, i.e.: ```markup $([System.DateTime]::Now) ``` is embedded in the .metaproj as the actually evaluated value, such as: ```markup 10/21/2020 4:36:30 AM ``` # Detect CI builds for every CI system In order to unify the environment variable to detect whether a build is a CI build as simply `CI` (as is done already in [GitHub Actions](https://docs.github.com/en/free-pro-team@latest/actions/reference/environment-variables), [Circle CI](https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables) and [GitLab](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)), you just place this bit of MSBuild in every project's `Directory.Build.props` to make things consistent everywhere: ```markup false true ``` # Modify all command-line builds in entire repo I used to place `msbuild.rsp` [response files](https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-response-files?view=vs-2019#msbuildrsp) alongside solution and project files to avoid repeating `-nr:false -v:m -bl` (no node reuse, minimal verbosity, binlogs). It was super annoying to have to have it all over the place. While [browsing another repo](https://github.com/microsoft/vs-streamjsonrpc/blob/master/Directory.Build.rsp), I just learned that since MSBuild 15.6+, you can now have a [Directory.Build.rsp response file](https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-response-files?view=vs-2019#directorybuildrsp) that affects every build in every descendent folder. This sounds like a great default one for me: ``` # See https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-response-files -nr:false -m:1 -v:m -clp:Summary;ForceNoAlign ``` Note I also prefer much more the `-` syntax rather than `/` for switches. # Write entire XML fragments in MSBuild with XmlPoke I recently needed to write an entire project file via a targets (weird, I know). I was dreading all the crazy angle brackets escaping as `<` and `>` when I decided to check the [latest official docs on XmlPoke](https://docs.microsoft.com/en-us/visualstudio/msbuild/xmlpoke-task?view=vs-2019) to refresh the parameters and format. To my surprise, I found this example: ```markup ``` Notice how the `$(Namespace)` property has **beautiful** XML inside. Which only makes sense, since, there had to be **some** advantage in MSBuild being XML, right? So I figured, if the namespaces can be a full nested XML element, could the `Value` be too? And the answer was a [resounding YEAHHH](https://github.com/kzu/SmallSharp/blob/main/src/SmallSharp/SmallSharp.targets#L88-L100): ```markup $(StartupFile) ``` That writes an entire `` element into a .user project file! For the attentive reader: you'd think I made a mistake, since, in XML-land, elements inherit the XML namespace of their parent element. Seems like the XML namespace management in XmlPoke is a bit on the loose side of things: the above snippets are in a .targets file without any xmlns (it's an SDK-style `` without namespace, to keep it clean. Yet, the property group is added without messing up the namespace: ```markup Program.cs ``` If it had inserted it without a namespace, per the XML spec, it should have added an `xmlns=""` to clear the parent Project node namespace. But alas, it didn't, and in this particular case, it makes it way more convenient this way :). I'm using this in [SmallSharp](https://github.com/kzu/SmallSharp) to initialize the .user project options properly with the right startup file for multi-startup top-level statements scripts in a single project :). # How to select first item in an ItemGroup I was entirely unaware of this [simple trick](https://gist.github.com/shadow-cs/cb5499b010bdacd1f778be29daf7f04c)! Turns out that when you assign a property value to an item metadata, and there are multiple items, all items are iterated and the property is assigned consecutively to each item metadata, leaving you with the last such item metadata as the property value. If you want the first item, you reverse the item group using the built-in item function and then assign the property: ```markup %(Reversed.Identity) ``` # How to include commit URL in nuget package description It's quite helpful to include a [direct link to the source code](https://www.nuget.org/packages/Microsoft.AspnetCore.Authorization) that produced a given package version, so your users can quickly see what's included (i.e. they may be tracking a bug fix). If you are using [Microsoft.SourceLink](https://www.nuget.org/packages?q=Microsoft.SourceLink), you already have all the information you need automatically populated for you. The following target will update the description just before the nuget is produced: ```markup $(Description) Built from $(RepositoryUrl)/tree/$(SourceRevisionId.Substring(0, 9)) $(RepositoryUrl) $(Description) ``` This will result in a short-ish link (we trim it to 9 chars, which is the common short sha in Git) to the repo, similar to many [ASP.NET Core](https://www.nuget.org/packages/Microsoft.AspNetCore.Http/) [packages](https://www.nuget.org/packages/Microsoft.AspNetCore.Mvc.Core/). If you're using [GitInfo](https://www.nuget.org/packages/GitInfo/) instead, you'll have to populate the `RepositoryUrl` yourself, and replace `SourceRevisionId` with `GitSha`. # How to include package reference files in your nuget This is a particularly common scenario if you're developing MSBuild tasks or Roslyn code analyzers: all the dependencies you use in your task, analyzer or source generator, need to be included in your nuget package too, alongside your project primary output (i.e. under the proper`analyzer, tools or build`folders). A very convenient way (if not comprehensive, since it won't include with transitive dependencies) to do so is by annotating your `PackageReference` themselves, like: ```markup ``` The `Pack` metadata value can then be used to include the assets in the package with the following target: ```markup @(NuGetPackageId -> Distinct()) @(PackageReferenceDependency -> '%(Pack)') <_PackageFiles Include="@(RuntimeCopyLocalItems)" PackagePath="$(BuildOutputTargetFolder)/$(TargetFramework)/%(Filename)%(Extension)" /> ``` Quite a few things to note in the above target that aren't too obvious: 1. We use the `RuntimeCopyLocalItems` item group which is resolved by `ResolvePackageAssets` and contains the stuff that is needed for the dependency to run (i.e. the actual binaries, not reference assemblies, if it includes them). 2. We use Inputs/Outputs on it so we can batch by `%(RuntimeCopyLocalItems.NuGetPackageId)`: this makes processing simpler inside the target, since we'll be dealing with a single `NuGetPackageId` for each batch, regardless of how many `RuntimeCopyLocalItems`there are. 3. We next get that package ID as a property, and find the `@(PackageReference)` with that ID, to determine if it needs to be packed or not. 4. Note we use property syntax next since we know there can be at most one such `@(PackageReferenceDependency)`, in which case we'd get either an empty value or `true` for `%(Pack)`. 5. If we have to pack, we include all the `@(RuntimeCopyLocalItems)` (in the current batch, MSBuild does this for us for free thanks to the Inputs/Outputs) and use as the package path the same as what the primary project output will use. These are added directly as `_PackageFiles` which is what the SDK-style project `Pack` uses. 6. We update the items metadata so they are also flagged as copy-local 7. Finally, the `ResolvedFileToPublish` is useful when creating dotnet tools, since packing those is slightly different and includes a publish operation. # How to build project when content files change Visual Studio will typically just report the project is up-to-date and doesn't need to be built if you just changed a `Content` or `None` item. If you want it to consider those file types to also trigger a build, just add the relevant items as `UpToDateCheckInput`: ``` ``` # How to launch multiple Azure Functions apps on different ports Turns out setting up `Hosts.LocalHostPort` via local.settings.json doesn't work (consistently at least, I keep getting semi-random *Value cannot be null. (Parameter 'provider')* errors on function startup), [contrary to what the documentation says](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash#local-settings-file). If you override the command line args directly, via a `launchSettings.json` , it works consistently and I even get faster startup ¯\_ (ツ)\_/¯: ```javascript { "profiles": { "api": { "commandName": "Project", "commandLineArgs": "start --port 7072" } } } ``` You might want to add a `--pause-on-error` in there too. # C# script function apps beyond Azure portal Creating a C# [function app in the Azure portal](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-azure-function) is incredibly simple and great for playing and learning. I love the simplicity and light-ness that comes from just having a single `.csx` script file with everything you need for the function. The more "serious" approach with a "proper" [C# project, the functions SDK](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-your-first-function-visual-studio) and the corresponding CI/CD setup seems quite the leap, in comparison. I [recently had to move](https://twitter.com/kzu/status/1300481855565725696) a bunch of functions from one subscription to another and wanted to take the chance to improve the maintainability for some in-portal functions I had. Moving them also failed in the portal, so I looking at a non-enjoyable time copying/pasting files over. Made me question why I used in-portal functions at all. So, while trying to save myself some time doing that, I got a zip of the existing in-portal function app using the Kudu debug console at [*https://\[APP\_NAME\].scm.azurewebsites.net/DebugConsole*](https://azdo-api.scm.azurewebsites.net/DebugConsole), and simply clicking the download icon next to the *wwwroot* folder: ![](/files/-MGzA_BfDpb1KUxyaVut) And that's where it clicked: why not just put all those files in my own [GitHub repo](https://github.com/kzu/azdo/tree/main/api) and deploy straight from there via a [subfolder](https://til.cazzulino.com/azure/publishing-function-app-from-github-folder)? ![](/files/-MGzB4qj1apabZ0fP_-_) Well, that \*totally\* works, and you can keep the simplicity of the `.csx` while having a proper deployment history (and rollback capabilities) you're likely going to need sooner or later, no matter how simple the function. I couldn't get rid of the *function.json* file though, so placing the Functions SDK attributes on the code in the .csx doesn't work, but I'd say it's an acceptable trade-off. # Publishing function app from GitHub folder For very simple functions, I really liked just hacking them in the Azure portal. This is super brittle, however, since changes that break the app can't be undone. Great for learning, not so much once you start depending on those functions. While you can deploy using [GitHub Actions](https://docs.microsoft.com/en-us/azure/azure-functions/functions-how-to-github-actions?tabs=javascript) and [Azure Pipelines](https://docs.microsoft.com/en-us/azure/azure-functions/functions-how-to-azure-devops?tabs=csharp), both seem a massive leap in complexity from the beauty of in-portal editing and updating. [Kudu](https://docs.microsoft.com/en-us/azure/app-service/deploy-continuous-deployment#option-1-kudu-app-service) (or *App Service build service*) is way simpler and generally sufficient for simple functions. You just select GitHub as the source: ![](/files/-MGz6jAF0OLIFAPaTFXe) And App Service build service as the build provider: ![](/files/-MGz71K2XL02xPSMXCVs) You next simply connect it to your GitHub repository and that's it. But what if you want to deploy a [subfolder](https://github.com/kzu/azdo/tree/main/redir) from the repository as the function app? Turns out you can just add an application setting to the function app, named `DEPLOYMENT_SOURCE`, pointing to the right subfolder (i.e. `.\redir` or `.\api` in my case) and that's it! Here you can see it in action in the logs, where just the `redir` subfolder is being sync'ed to the `wwwroot` for the function app: ![](/files/-MGz7egi1TfoEC7uKfzS) # Exploring Azure Data with Kusto and Dashboards In order to more effectively learn [Kusto](https://docs.microsoft.com/en-us/azure/azure-monitor/log-query/query-language) (the query language powering Azure analytics, log querying and PowerBI) and data visualization capabilites in Azure, I did the following: 1. [Created a cluster and database](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) (this takes quite a while) 2. Open the [Azure Data Explorer](https://aka.ms/kwe) (a.k.a. Kusto Web Explorer) at 3. Add the cluster with the full URI or alternatively just the `name.region` parts (i.e. `kzukusto.centralus` vs [`https://kzukusto.centralus.kusto.windows.net`](https://kzukusto.centralus.kusto.windows.net)) 4. Optionally add an arbitrary Application Insights app as a "virtual cluster", [using a url with the format](https://docs.microsoft.com/en-us/azure/data-explorer/query-monitor-data#connect-to-the-proxy) `https://ade.applicationinsights.io/subscriptions//resourcegroups//providers/microsoft.insights/components/` 5. Right--lick database and select Ingest new data ![Ingest new data context menu](/files/-MDx98w1WH2Zx_GWtvr9) 6. Find some interesting [Azure Open Dataset from the catalog](https://azure.microsoft.com/en-us/services/open-datasets/catalog/) that has an Azure storage URL readily available from the Azure Open Datasets catalog, such as the [Bing COVID-19 Data](https://azure.microsoft.com/en-us/services/open-datasets/catalog/bing-covid-19-data/) (I used the `.jsonl` link). NOTE: the `.json` one will not properly infer schema because it has a root object of type array. The `.jsonl` is actually a "JSON fragment" (if that even exists, would be the equivalent of an XML fragment) where each entry is just a JSON entry/line in the file. ![](/files/MdceBwuvfpTqwtmnl1Xo) The JSON version is preferable to `.csv` because it properly infer the data type for columns. 7. Click on `Dashboards` for the new stuff here. Parameters driven by queries are particularly handy: ![Query-driven multi-select parameter for widget](/files/-MDxBsQzMX0V4f9OGXH_) While editing the query/widget, if the parameter is used in the query, you can interactively change its value to explore the visualizations. For example: ![Used parameter becomes enabled for selection](/files/-MDxD77EKTw37D4YsCx5) Whereas if it wasn't used: ![Unused parameter unavailable](/files/-MDxDUyIDDTM_pU3lv0K) After shaping the `Results`the way you want, clicking the Visual tab allows configuring a bunch of visualizations. Inference works quite nicely if the data/results are filtered to just what you want to display. ![Many chart options and inference that works great](/files/-MDxEFQhX1FPsRDMdYhk) # Shared secret authorization with Azure SignalR Service While testing out [Azure Functions development and configuration with Azure SignalR Service](https://docs.microsoft.com/en-us/azure/azure-signalr/signalr-concept-serverless-development-config#negotiate-experience-in-class-based-model), I needed a very simple key-based (shared secret) authorization mechanism so that my console-based SignalR client could connect to my Azure SignalR Service-powered hub using a very simple mechanism. The docs and the sample showcased the `negotiate` endpoint returning the connection info directly: ``` [FunctionName("negotiate")] public SignalRConnectionInfo Negotiate([HttpTrigger(AuthorizationLevel.Anonymous)]HttpRequest req) ``` But a [stackoverflow answer](https://stackoverflow.com/a/55586165/24684) pointed me to the solution: just the proper `IActionResult` using `OkObjectResult` with the connection info when the access key is properly validated: ``` [FunctionName("negotiate")] public IActionResult Negotiate( [HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequest req, [SignalRConnectionInfo(HubName = "events")] SignalRConnectionInfo connectionInfo) { var expectedKey = Environment.GetEnvironmentVariable("AccessKey"); if (string.IsNullOrEmpty(expectedKey)) return new OkObjectResult(connectionInfo); var accessKey = req.Query["accessKey"]; if (StringValues.IsNullOrEmpty(accessKey) || !StringValues.Equals(expectedKey, accessKey)) return new UnauthorizedResult(); return new OkObjectResult(connectionInfo); } ``` # Using Azure File Copy from DevOps yaml pipeline I learned that it's not enough to authorize Azure Resource Manager access from DevOps Oh boy, did I waste time on this one :(. So I had my pipeline pretty naively doing an upload to blob storage: ``` - task: AzureFileCopy@4 displayName: Upload Vsix inputs: SourcePath: '$(Pipeline.Workspace)\vsix\RoslynDeployment.$(Build.BuildId).vsix' azureSubscription: 'roslyn-Azure' Destination: 'AzureBlob' storage: 'roslyn' ContainerName: 'vsix' BlobPrefix: '$(Build.SourceBranchName)/RoslynDeployment.$(Build.BuildId).vsix' ``` I used a service principal [managed by DevOps which is the recommended approach](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops#create-an-azure-resource-manager-service-connection-using-automated-security). The blob storage account was under the same subscription, where the automatically created app properly showed up in IAM: ![Access control (IAM) pane for storage account](/files/-M9nrbiYTx2HwJZmmjdD) as a contributor: ![DevOps-managed app as contributor to the storage account](/files/-M9nsIQM-CBl30nDjcBt) I kept getting a 403 response when the task run, with the message `This request is not authorized to perform this operation using this permission.` Turns out being a **Contributor** is not enough. I tried [changing guest user permissions](https://docs.microsoft.com/en-us/azure/devops/pipelines/release/azure-rm-endpoint?view=azure-devops#insufficient-privileges-to-complete-the-operation), but in the end the only thing that worked was manually adding the [Storage Blob Data Contributor role](https://github.com/MicrosoftDocs/azure-docs/issues/36454), which I found mentioned in a[ blog post](https://www.catrina.me/azcopy-403-error/). In the process I learned how DevOps creates the app registration and what-not, but still, not fun. [Submitted a doc fix](https://github.com/MicrosoftDocs/azure-devops-docs/pull/8622) for the [AzureFileCopy task docs](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-file-copy-version3?view=azure-devops) so this is more easily discoverable. # Code-less redirection with serverless Azure Functions How to quickly and simply configure redirections without writing code in Azure Functions Say you want to have a nicer URI for something (like an Azure storage blob, a feed or something else). You likely have a nice short custom domain (i.e. I use kzu.io for things like this), and would like to set up arbitrary (temporary or permanent) redirections. This can trivially be achieved by creating an empty Functions App and leveraging [Functions Proxies](https://docs.microsoft.com/en-us/azure/azure-functions/functions-proxies). The `proxies.json` file for code-less redirects looks as follows: ```javascript { "$schema": "http://json.schemastore.org/proxies", "proxies": { "[SOME_ID]": { "matchCondition": { "methods": [ "GET" ], "route": "[SHORT_PATH_HERE]" }, "responseOverrides": { "response.statusCode": "[301|302|307|308|", "response.headers.location": "[LONG_URL_HERE]" } } } } ``` You can have as many of those IDs/entries as needed. For example, this is one I use to set up `https://pkg.kzu.io/index.json` > `https://kzu.blob.core.windows.net/nuget/index.json`: ```javascript { "$schema": "http://json.schemastore.org/proxies", "proxies": { "default": { "matchCondition": { "methods": [ "GET" ], "route": "index.json" }, "responseOverrides": { "response.statusCode": "307", "response.headers.location": "https://kzu.blob.core.windows.net/nuget/index.json" } } } } ``` # How to run Azure Storage unit tests in CI If you have tests that need to exercise Blob, Queue or Table storage from Azure Storage, you can use [Azurite](https://github.com/Azure/Azurite) (v3) in CI, which can be configured for GitHub Actions as follows: ``` - name: ⚙ azurite run: | npm install azurite npx azurite-table & ``` The first line will install it, and the second will start it in a background process. Tests will need to use `CloudStorageAccount.DevelopmentStorageAccount` to access the locally running instance automatically. Note that this is compatible with the older Azure Storage Emulator included with the Azure SDK with Visual Studio, so no changes are needed between local test runs and CI. You can see this in action in the following run: which is running on all supported .NET platforms via the workflow definition at . # How to skip steps or jobs in GitHub Actions for PRs from forks [Encrypted secrets in GitHub](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow) actions aren't available for builds from forks, so if your build script includes PRs, like [Avatar](https://github.com/kzu/avatar/blob/main/.github/workflows/build.yml#L2-L6): ```yaml on: push: branches: [ main, dev, 'feature/*', 'rel/*' ] pull_request: types: [opened, synchronize, reopened] ``` You may need to limit steps or entire jobs from running when PRs are coming from forks due to missing secrets (i.e. pushing an output package to a CI nuget feed, say). The "magic" thing is to use an `if` condition with the following expression: ```yaml push: name: push nuget.ci runs-on: ubuntu-latest needs: [build, acceptance] if: ${{ !github.event.pull_request.head.repo.fork }} steps: ``` Note that in the above case, an entire job is skipped, but you can also apply it to a step instead. # Update version and publish npm from GH Setting up CI with GitHub Actions to update a node.js package version from GitHub release tag and publish it to npm Super proud of this, since it's my first node.js package ever (created brand-new account on and all 😁), for doing [syntax highlighting](https://github.com/dotnetconfig/highlightjs-dotnetconfig) for [dotnetconfig](https://dotnetconfig.org/) that works in [docfx](https://github.com/dotnetconfig/dotnet-config/tree/dev/docs). Assuming you already have a `package.json` in the root repo dir, add the `.github\workflows\npm.yml` as follows: ``` name: publish on: release: types: [created] jobs: publish-npm: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - uses: actions/setup-node@v1 with: node-version: 12 registry-url: https://registry.npmjs.org/ - name: Set version from tag run: npm --no-git-tag-version version ${GITHUB_REF#refs/*/} - run: npm publish env: NODE_AUTH_TOKEN: ${{secrets.npm_token}} ``` 1. I'm running only on release creation 2. Using the GitHub envvar for the [checked out tag](https://stackoverflow.com/questions/58177786/get-the-current-pushed-tag-in-github-actions/58178121#58178121), I'm [updating the version](https://docs.npmjs.com/cli/version) but opting out of the git behavior (which fails on CI because it's a detached head at that point) 3. Finally you'll need to create that secret in [GH](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets) for the publish step. # Push to protected branch from GitHub actions It turns out that you really can't just `git push` from your GitHub actions [if the repository has branch protection turned on](https://github.community/t/how-to-push-to-protected-branches-in-a-github-action/16101) or required checks before merging. Sorta makes sense, but still a PITA. The solution that worked for me was to [use a different token on checkout](https://github.community/t/how-to-push-to-protected-branches-in-a-github-action/16101/34). Since the awesome GitHub CLI [allows using a separate, higher-permissions token](https://github.com/cli/cli/issues/1229) named `GH_TOKEN` (since depending on the command you use, you might need a different one than `GITHUB_TOKEN`), I decided to (ab)use the same: An [example workflow](https://github.com/devlooped/oss/blob/main/.github/workflows/changelog.yml) that uses this to generate a full changelog and push it to main on releases looks like this: ``` name: changelog on: release: types: [released] env: GH_TOKEN: ${{ secrets.GH_TOKEN }} jobs: changelog: runs-on: ubuntu-latest steps: - name: 🔍 GH_TOKEN if: env.GH_TOKEN == '' env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: echo "GH_TOKEN=${GITHUB_TOKEN}" >> $GITHUB_ENV - name: 🤘 checkout uses: actions/checkout@v2 with: fetch-depth: 0 ref: main token: ${{ env.GH_TOKEN }} - name: ⚙ changelog uses: faberNovel/github-changelog-generator-action@master with: options: --token ${{ secrets.GITHUB_TOKEN }} --o changelog.md - name: 🚀 changelog run: | git config --local user.name github-actions git config --local user.email github-actions@github.com git add changelog.md git commit -m "🖉 Update changelog with ${GITHUB_REF#refs/*/}" git push ``` Important parts: * I default the `GH_TOKEN` envvar to a same-name secret, if present * If it's not present, I default it to `GITHUB_TOKEN` * Checkout always uses `GH_TOKEN`, which now may be a higher-permissions PAT than the default * I do the defaulting since the push **will** succeed if the repository doesn't use branch protection for `main` and in that case I don't want to always force the presence of a `GH_TOKEN` secret.