Indexing and searching NuGet.org with Azure Functions and Search - .NET fwdays'20 online conference

Editor's Notes

• #2: https://pixabay.com
• #5: Show feature in action in Visual Studio (and show you can see basic metadata etc.)
• #6: Copied in 2017 in VS - https://www.hanselman.com/blog/VisualStudio2017CanAutomaticallyRecommendNuGetPackagesForUnknownTypes.aspx Demo the feed quickly?
• #7: More and more packages OData feed slow to query & scheduled for deprecation Is there a better way?
• #8: More and more packages OData feed slow to query & scheduled for deprecation Is there a better way?
• #10: Demo: click around in the API to show some base things
• #12: Raw API - click around in the API to show some base things, explain how a cursor could go over it Root https://api.nuget.org/v3/catalog0/index.json Page https://api.nuget.org/v3/catalog0/page0.json Leaf https://api.nuget.org/v3/catalog0/data/2015.02.01.06.22.45/adam.jsgenerator.1.1.0.json Explain CatalogDump NuGet.Protocol.Catalog comes from GitHub CatalogProcessor feches all pages between min and max timestamp My implementation BatchCatalogProcessor fetches multiple pages at the same time and build a “latest state” – much faster! Fetches leaves, for every leaf calls into a simple method Much faster, easy to pause (keep track of min/max timestamp)
• #17: Will use storage queues n demo’s to be able to run things locally. Ideally use SB topics or event grid (transactional)
• #18: Create a new TimerTrigger function We will need a function to index things from NuGet Timer will trigger every X amount of time Timer provides last timestamp and next timestamp, so we can run our collector for that period Snippet: demo-timertrigger Mention HttpClient not used correctly: not disposed, so will starve TCP connections at some point Go over code example and run it var httpClient = new HttpClient(); var cursor = new InMemoryCursor(timer.ScheduleStatus?.Last ?? DateTimeOffset.UtcNow); var processor = new CatalogProcessor( cursor, new CatalogClient(httpClient, new NullLogger<CatalogClient>()), new DelegatingCatalogLeafProcessor( added => { log.LogInformation("[ADDED] " + added.PackageId + "@" + added.PackageVersion); return Task.FromResult(true); }, deleted => { log.LogInformation("[DELETED] " + deleted.PackageId + "@" + deleted.PackageVersion); return Task.FromResult(true); }), new CatalogProcessorSettings { MinCommitTimestamp = timer.ScheduleStatus?.Last ?? DateTimeOffset.UtcNow, MaxCommitTimestamp = timer.ScheduleStatus?.Next ?? DateTimeOffset.UtcNow, ServiceIndexUrl = "https://api.nuget.org/v3/index.json" }, new NullLogger<CatalogProcessor>()); await processor.ProcessAsync(CancellationToken.None);
• #19: Each function should only do one thing! We are violating this.
• #20: https://github.com/thinktecture/azure-functions-extensibility
• #21: Go over Approach2 code Show this is MUCH simpler – trigger binding that provides input, queue output bindign to write that input to a queue Let’s go over what it takes to build a trigger binding NuGetCatalogTriggerAttribute – the data needed for the trigger to work – go over properties and attributes Hooking it up requires a binding configuration – NuGetCatalogTriggerExtensionConfigProvider It says: if you see this specific binding, register it as a trigger that maps to some provider So we need that provider – NuGetCatalogTriggerAttributeBindingProvider Provider is there to create an object that provides data. In our case we need to store the NuGet catalog timestamp cursor, so we do that on storage, and then return the actual binding – NuGetCatalogTriggerBinding In NuGetCatalogTriggerBinding, we have to specify how data can be bound. What if I use a differnt type of object than PackageOperation? What if someone used a node.js or Python function instead of .NET. Need to define the shape of the data our trigger provides. PackageOperationValueProvider is also interesting, this provides data shown in the portal diagnostics CreateListenerAsync is where the actual triger code will be created – NuGetCatalogListener NuGetCatalogListener uses the BatchCatalogProcessor we had previously, and when a package is added or deleted it will call into the injected ITriggeredFunctionExecutor ITriggeredFunctionExecutor is Azure Functions framework specific, but it’s the glue that will clal into our function with the data we provide Note StartAsync/StopAsync where you can add startup/shutdown code ONE THING LEFT THAT IS NOT DOCUMENTED – Startup.cs to register the binding. And since we are in a different class library, also need Microsoft.Azure.WebJobs.Extensions referenced to generate \bin\Debug\netcoreapp2.1\bin\extensions.json As a result our code is now MUCH cleaner, show it again and maybe also show it in action Mention [Singleton(Mode = SingletonMode.Listener)] – we need to ensure this binding only runs single-instance (cursor clashes otherwise). This is due to ho the catalog works, parallel processing is harder to do. But we can fix that by scaling the Indexer later on. Show Approach3 PopulateQueueAndTable Same code, but a bit more production worthy Sending data to two queues (indexing and downloading) Storing data in a table (and yes, violating “do one thing” again but I call it architectural freedom)
• #22: Next up will be downloading and indexing. Let’s start with downloading. Grab a copy of the .nupkg from NuGet and store it in a blob Redundancy - no need to re-download/stress NuGet on a re-index
• #23: Go over Approach3 code DownloadToStorage uses a QueueTrigger to run whenever a message appears in queue Note no singleton: we can scale this across multiple instances/multiple servers Uses a Blob input binding that provides access to a blob Note the parameters, name of the blob is resolved based on data from other inputs which is prety nifty Our code checks whether it’s an add or a delete, and either downloads+uploads to the blob reference, or delets the blob reference
• #24: Next up will be indexing itself. There are a couple of things here…
• #26: Go over Approach3 code PackageIndexer uses a QueueTrigger to run whenever a message appears in queue Uses a Blob input binding that provides access to a blob where we can write our indexed entity – will show this later Based on package operation, we will add or delete from the index RunAddPackageAsync has some plumbing, probably too much, to dowload the .nupkg file and store it on disk Note: we store it on disk as we need a seekable stream. So why no memoy stream? Some NuGet packages are HUGE. Find PEReader usage and show how it will index a given package’s public types and namespaces All goes into a typeNames collection. Now: how do we add this info to the index? Show PackageDocument class, has MANY properties First important: the Identifier property has [Key] applied. Azure Search needs a key for teh document so we can retrieve by key, which could be useful when updating existing content or to find a specific document and delete it from the index. Second important: TypeNames is searchable. Also mention “simpleanalyzer”: “Divides text at non-letters and converts them to lower case.” Other analyzers remove stopwords and do other things, this one should be as searchable as possible. Other fields are sometimes searchable, sometimes facetable – a bit of leftover from me thinking about search use cases. The R# API ony searches on typename so could make everything else just retrievable as well. Of course, index is not there by default, so need to create it. We do this when our function is instantiated (static constructor, so only once per launch of our functions) Is this good? Yes, because only once per server instance our function runs on. No because we do it at one point, what if the index is deleted in between and needs to be recreated? Edge case, but a retry strategy could be a good idea... Next, we create our package document, and at one point we add it to a list of index actions, and to blob storage indexActions.Add(IndexAction.MergeOrUpload(packageToIndex)); JsonSerializer.Serialize(jsonWriter, packagesToIndex); Writing to index using batch - var indexBatch = IndexBatch.New(actions); Leftover code from earlier, batch makes no sense for one document, but in case you want to do multiple in one go this is the way. Do beware a batch can only be several MB in size, for this NuGet indexing I can only do ~25 in a batch before payload is too large. That’s… it! Run approach 3 (for last hour) and see functions being hit / packages added to index Go to Azure Search portal as well, show how importer would work in case of fire
• #28: Now we need to make ReSharper talk to our search. We have the index, so that should be a breeze, right?
• #29: Go over Web code RunFindTypeApiAsync and RunFindNamespaceAsync Both use “name” as their query parameter to search for RunInternalAsync does the heavy lifting Grabs other parameters Runs search, and collects several pages of results Why is this ForEachAsync there? Search index has multiple versions for every package id, yet ReSharper expects only the latest matching all parameters Azure Search has no group by / distinct by, so need to do this in memory. Doing it here by fetching a maximum number of results and doing the grouping manually. Use the collected data to build result. Add matching type names etc. Example requests: http://localhost:7071/api/v1/find-type?name=JsonConvert http://localhost:7071/api/v1/find-type?name=CamoServer&allowPrerelease=true&latestVersion=false https://nugettypesearch.azurewebsites.net/api/v1/find-type?name=JsonConvert In ReSharper (devenv /ReSharper.Internal, go to NuGet tool window, set base URL to https://nugettypesearch.azurewebsites.net/api/v1/) Write some code that uses JsonConvert / JObject and try it out.