Blog posts in category: BigData

Microsoft Azure's Monitor for Azure Functions is buggy - check the logs for any missing data

by Patrick Lee on 28 Jan 2023 in categories tech BigData with tags Azure AzureFunctions Application Insights

In the Azure portal, for Azure Functions, I've found that the Monitor section for individual subfunctions is not infrequently a bit buggy, in that it misses out some function runs, particularly for functions trigged by a timer. To check whether the function actually ran on a particular day, check the logs for the function: they are usually far more complete.

Want to move large files/large numbers of files to/from Azure Storage? Use AzCopy or Microsoft Azure Storage Data Movement

by Patrick Lee on 20 May 2022 in categories tech BigData with tags AzureFunctions AzureStorage

If you need to move thousands of Azure blobs, or very large ones (many GB) then you can either use AzCopy (a command line tool which you run locally), or if you want to do things programmatically, you can use the Microsoft Azure Storage Data Movement library. That library is ...

How to extend the timeout for Azure functions beyond the default limits - use AzureFunctionsJobHost__functionTimeout setting

by Patrick Lee on 20 May 2022 in categories tech BigData with tags AzureFunctions

By default, Azure functions timeout after 10 minutes (or 30 minutes if running in an App Service Plan, or a Premium plan). The official documentation (see https://docs.microsoft.com/en-us/azure/azure-functions/functions-host-json#functiontimeout) says that you can increase (or reduce) this by adding a functionTimeout value to host.json. But this doesn't seem to work for a ...

Working towards a Pensions Dashboard

by Patrick Lee on 08 May 2022 in categories actuarial BigData pensions tech with tags dashboard pensionsdashboard

I am creating a prototype, suitable for use with the UK's Pensions Dashboard project.  Early days, but have made good progress so far!  More in due course.

How to trigger and query an Azure Entity Function without using an ordinary function - just use the built in HTTP webhooks!

by Patrick Lee on 08 May 2022 in categories tech BigData with tags AzureFunctions Functions DurableFunctions

A simpler way to create and query Azure Entity Functions (previously called Durable Entities): just use the built in webhooks.

Azure Durable Functions: not necessarily suitable for orchestrating large numbers of tasks

by Patrick Lee on 20 Apr 2021 in categories tech BigData with tags Azure SErverless Functions

Microsoft's Durable Functions (see https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp ) are a very useful tool for performing some complex tasks in a robust serverless way. But in situations where there is a large number of different tasks involved, they introduce their own overhead which can slow things down significantly. This is because if there ...

.NET Core 3.0 release is planned for September 2019!

by Patrick Lee on 05 Jul 2019 in categories tech BigData with tags Blazor ASP.NET Core

Further to my previous post about Blazor see here ) I am excited about the forthcoming release of version 3.0 of .NET Core by Microsoft, expected in September 2019. In particular, not only will this version include Blazor, but it will include support for machine learning. https://msdn.microsoft.com/en-us/magazine/mt848631.aspx says: NET Core ...

I just completed Microsoft's Reinforcement Learning Explained course

by Patrick Lee on 01 Jul 2019 in categories tech BigData with tags Python AI lifelong learning Jupyter notebooks CNTK

Yesterday I completed what I have found to be the most demanding course so far on the Microsoft Professional Program for Artificial Intelligence , Reinforcement Learning Explained . It took me 66.5 hours (more than twice as long as previous edX courses have taken me) and my final mark was ...

June 2019: I've been re-elected to IFoA Council - thank you!

by Patrick Lee on 28 Jun 2019 in categories tech actuarial BigData with tags AI

Further to this post , I'm very pleased to have been re-elected to the Council of the Institute and Faculty of Actuaries for what will now be my third term (for 3 years from 2019 to 2022). Many thanks to all those who supported me, and indeed all those who ...

I just completed Microsoft's Computer Vision and Image Analysis course

by Patrick Lee on 08 Jun 2019 in categories tech BigData with tags Python AI lifelong learning Jupyter notebooks CNTK

I recently completed the above Microsoft/edX course (full details of the course are here ) with a final mark of 92%. This was my 8th course on the Microsoft Professional Program for Artificial Intelligence, I plan to do my 9th course ( Reinforcement Learning Explained ) during the rest of ...

Next