Patrick is on his hands and knees, examining the appliance with the focused scepticism of a claims assessor who has seen this sort of thing before. Is it genuinely broken? Or is it just having a bad day? These are different things, and they have very different reserving implications.

The Post-it note on the front reads "DOD: TBC — monitor." Patrick has seen appliances stage miraculous recoveries before. Premature settlement would be irresponsible.

Pucktick, for his part, has found the puddle and is perfectly happy.

The Actuaries in the Wild series follows actuarial thinking as it escapes the office and colonises everyday life — holidays, kitchens, birthday parties, and anywhere else a risk register has no business being. Previous entry: #8 Holiday Risk Assessment.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption | #7 Excel Relapse | #8 Holiday Risk Assessment | #9 Confident Nonsense | #10 Nearly There | #11 Best Estimate

How many graduate actuarial positions are available in the UK for September 2026? Is hiring up, down, or flat compared to last year?

These seem like straightforward questions. I spent a day trying to answer them. What I found — and what I got wrong — is itself worth reporting.

How many graduate positions are there?

The Actuarial Profession Guide 2025/26 (37th edition, published annually by actuarialcareers.co.uk in partnership with the IFoA) is the closest thing to a definitive answer. Its Employer Directory lists 32 firms recruiting actuarial trainees, and most publish their graduate intake numbers — from 1 (Atkin Pensions) to 100+ (Aon). A rough sum of the specific numbers gives approximately 400–500 graduate positions across the listed firms, though 12 of the 32 say “Varies” rather than giving a number. The guide does not attempt an industry-wide total, and smaller employers and non-traditional hirers are not included. What follows below is my own independent research, which found 20 employers — some overlapping with the guide, some not.

The IFoA publishes total membership (34,531 as at 2024/25, of whom 15,380 are students) but not how many new students register each year, or how many graduate positions employers collectively offer. Job boards count listings, not positions — and their numbers vary wildly depending on how broadly they define “actuarial graduate” (Glassdoor shows 13 in April 2026; LinkedIn shows 545 for ostensibly the same search).

So alongside the guide’s data, here is a verified, sourced snapshot of every employer I could independently confirm is running a graduate actuarial scheme for September 2026.

20 employers confirmed for September 2026

I checked career pages, graduate aggregators (Bright Network, Gradcracker, Prospects, StudySmarter), and specialist actuarial job boards. The following employers have confirmed September 2026 actuarial graduate schemes, verified from their own career pages or structured listings on aggregator sites.

Consultancies

Big 4

Insurers

Lloyd's market, banking, reinsurance & regulation

Additional employers visible on Bright Network but behind a login wall: Marsh, SCOR (x2 listings), and an additional L&G listing.

What about Lloyds Banking Group?

Lloyds Banking Group runs an actuarial graduate scheme offering £42,000 — the highest published starting salary of any scheme I found. However, their careers page currently shows "This opportunity is now closed for applications in 2025" and invites candidates to register interest. I could not confirm a 2026 intake has been announced. The scheme itself is notable for its 4-year duration and emphasis on Python, R, and Prophet alongside traditional actuarial work.

Source: Lloyds Banking Group careers

What do they pay?

Seven employers publish their starting salaries explicitly. The range is £28,332 (Vitality, Bournemouth) to £38,000 (Lloyd's of London, London).

For comparison, here is what the British Army pays new recruits (from the official Army careers page):

A graduate entering the Army as an officer earns £34,676 during training and £41,456 on commissioning — comparable to or above most actuarial graduate schemes, with the Army also providing free accommodation, healthcare, pension contributions, and 30 days' leave plus bank holidays. A non-graduate joining the Intelligence Corps reaches £34,083 as a Lance Corporal after Phase 2 training — more than some actuarial graduate starting salaries. And here is what a newly qualified teacher earns (from Get Into Teaching, GOV.UK, as at September 2025):

Teachers also receive a defined benefit pension with employer contributions of 28.6% — substantially more generous than most private sector pension schemes. A typical teacher reaches at least £45,352 after 5 years.

These comparisons matter: actuarial employers are competing for numerate graduates against a much wider market than just other actuarial firms.

For broader context, the Hays UK Salary & Recruiting Trends 2025 Guide (published via ActuarialCareers.co.uk) provides market averages for graduate actuaries by region and sector:

Source: Hays UK Salary & Recruiting Trends 2025 Guide (GI and Pensions); 2024 Guide (Life)

These are averages across the market, not specific to any employer. The London premium is significant — around £6,000–£11,000 above regional rates. Pensions consulting pays less at graduate level than insurance, though the gap narrows with seniority.

Every scheme I checked offers full IFoA exam sponsorship: paid study leave, exam fees, study materials, and tutorials. Several (Zurich, Beazley, WTW) explicitly offer salary increases with each exam passed.

What I missed — and what's still missing

My original version of this article claimed that no employer publishes graduate cohort sizes. That was wrong. The Actuarial Profession Guide, published annually for nearly 40 years, includes intake numbers for most of its 32 listed employers. I missed this definitive resource, and I’m grateful to Andrew Watcham FIA (Director and Scheme Actuary, Isio) for pointing it out.

What does remain genuinely difficult is constructing an industry-wide total. The guide covers 32 employers but the profession has many more — including the Big 4 (PwC, Deloitte, EY, KPMG), several Lloyd’s market firms, and specialist insurers, none of which appear in the guide’s directory. My research found 11 employers not in the guide. Neither source is complete.

Year-on-year comparison should be possible using previous editions of the guide, which have been published since 1988/89. I plan to source earlier editions and analyse how graduate intake numbers have changed over time — particularly whether AI adoption is affecting hiring volumes. That analysis will follow in a separate article.

The IFoA pipeline — what the admissions data tells us

The IFoA doesn't routinely publish annual student admissions data in an accessible format. Its Facts and Figures page shows total membership (34,531 as at 2024/25, of whom 15,380 are students) but these are stock figures — how many students exist at a point in time, not how many new students join each year.

However, the IFoA used to publish annual admissions figures in its annual reports, and did so again in its 2023/24 annual report. Here is the available time series:

This tells us that approximately 2,500–3,500 people registered as new IFoA students each year over the period covered. The 2023/24 figure of 3,088 is within the historical range — but that figure pre-dates the rapid advances in AI capability during 2024-2026. If AI is changing the volume or nature of graduate actuarial hiring, the effect would show up in 2024/25 and 2025/26 admissions data — which the IFoA has not yet published.

But new student admissions are not the same thing as UK graduate positions. The IFoA is a global qualification body — over half of its students are based outside the UK, with large student populations in Ireland, India, and elsewhere. Some of those international students will be working in actuarial roles locally; others may be studying independently.

In the UK, the standard route into the profession is a full-time paid graduate role with an actuarial employer. These are not academic studentships — they are salaried positions (£28,000–£38,000 as we've seen) where you work as an actuarial analyst or trainee consultant while studying for IFoA exams alongside the day job. Employers typically provide a study package: paid study leave (a set number of half-days per exam subject, up to an annual maximum), exam fees, tutorial courses, and study materials. Qualifying as a Fellow of the IFoA typically takes around 7 years for those who complete the process — and roughly half of those who start do not reach Fellowship (based on my own analysis of enriched IFoA qualifier data). This employer-sponsored model is so dominant in the UK that "actuarial student" and "trainee actuary in a paid role" are essentially synonymous.

One statistic underlines how far the IFoA student population is from the "graduate trainee" image. The IFoA's 2024/25 annual report gives the "average age" of its students as 34.9. That is a surprisingly high figure for a category that in the UK is dominated by people in their twenties working towards qualification. The IFoA does not define whether this is a mean or median, and does not break it down by region. If it is being driven by a large cohort of older international students — or by long-standing students who registered years ago and are still on the books — it would be helpful for the IFoA to explain the composition behind this number.

The 3,088 admissions figure therefore includes a mix of: UK graduates starting these employer-sponsored roles (which is what this article is about), international candidates in equivalent roles abroad, career changers entering the profession mid-career, and some university students registering early to gain exam exemptions. The number of UK graduate actuarial positions in any given year is some fraction of these total admissions, but nobody publishes what that fraction is.

This is a gap worth closing. Actuarial science graduates, career changers, and employers would all benefit from knowing the approximate size of the UK graduate actuarial job market. The IFoA, as the professional body, is best placed to publish this — ideally with a UK/international breakdown.

What the picture tells us

Even without hard numbers, the qualitative picture is clear:

All major employer categories are hiring. Big 4 (all four confirmed), consultancies (Aon, WTW, LCP, First Actuarial, Broadstone), insurers (Aviva, L&G, Phoenix, Zurich, Just, Vitality, Beazley), Lloyd's market (Lloyd's of London, Arch Capital), reinsurance (SCOR), banking (Lloyds Banking Group — probable but unconfirmed for 2026), and public sector (GAD, FCA). But "all the same employers are still running schemes" does not mean they are hiring the same number of people. Cohort sizes could be shrinking — particularly as AI tools take on more of the routine analytical work that graduate trainees have traditionally done — and we would have no way of knowing from public data.

Most schemes closed between October 2025 and February 2026. If you're reading this in April 2026 and haven't yet applied, the main recruitment cycle is over. A few positions remain open on a rolling basis (First Actuarial, LCP, Broadstone, Phoenix/Standard Life, Vitality). The lesson for students: applications for September 2027 schemes will likely open from September 2026.

Starting salaries cluster between £31,000 and £38,000 for most employers, with a clear London premium. Outliers at both ends: Vitality at £28,332 (Bournemouth) and Lloyd's of London at £38,000 (London). If Lloyds Banking Group's £42,000 is confirmed for a 2026 intake, it would be the highest by some margin.

Methodology and sources

This research was conducted on 4 April 2026 using a combination of direct employer career page checks, graduate aggregator searches (Bright Network, Gradcracker, Prospects.ac.uk, StudySmarter, ActuarialCareers.co.uk), and web searches. Salary data comes from either employer-stated figures on career pages or the Hays UK Salary & Recruiting Trends Guide (2024/2025 editions) published via ActuarialCareers.co.uk. IFoA membership data is from the IFoA Annual Report 2024/25; student admissions data is from the IFoA Annual Reports 2015/16 through 2018/19 and the 2023/24 Annual Report. All sources are linked inline.

I have not included employers where I could not independently verify a 2026 scheme from a primary or structured secondary source. Several employers (Mercer, Hymans Robertson, Barnett Waddingham, XPS Pensions) almost certainly run 2026 graduate schemes, but their career pages showed only generic programme descriptions without year-specific confirmation at the time of research.

If your employer runs a graduate actuarial scheme and is not listed here, I would welcome corrections — please contact me via INQA Group.

Update (6 April 2026): The primary resource I should have consulted is the Actuarial Profession Guide 2025/26 (37th edition, published by actuarialcareers.co.uk in partnership with the IFoA), which lists 32 employers with graduate intake numbers and locations. I was not aware of this guide when I published the original article.

Patrick Lee is the founder of INQA Group, a professional community helping actuaries leverage AI and modern tools. He qualified as a Fellow of the Institute of Actuaries in 1990 and worked in actuarial consulting, pensions, and software development before founding INQA Group.

The plumber working under the kitchen sink and the crack running up the hallway wall are, of course, short-term noise. The methodology is sound. The fact that it has been wrong every year is not a flaw — it is a feature. Consistency of approach matters more than consistency with reality.

The Treasury understands this perfectly.

The Assumption Theatre series explores the dark art of actuarial assumption-setting — the pretence of objectivity, the negotiation, and the uncomfortable truth that enormous decisions rest on numbers someone essentially chose. Previous entry: #6 Long-Term Assumption.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption | #7 Excel Relapse | #8 Holiday Risk Assessment | #9 Confident Nonsense | #10 Nearly There

For most people, this is a perfectly normal exchange. For Patrick, it's an unresolved ambiguity. Nearly there geographically? Temporally? Or is there some unstated confidence interval he hasn't been briefed on?

The sat nav knows exactly how far it is. Beth knows approximately how far it is. Patrick needs to know which kind of knowing they're doing.

The Actuarial Pedantry series explores the domestic consequences of a brain that cannot switch off. Previous entries: #5 Define Everyone.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption | #7 Excel Relapse | #8 Holiday Risk Assessment | #9 Confident Nonsense

Every actuary has worked with a Preston. He doesn't understand the methodology, hasn't read the assumptions, and couldn't tell you the difference between a best estimate and a point estimate. But he knows confidence when he sees it. And the AI has confidence in spades.

The actuary, meanwhile, knows that a model proclaiming 99.7% confidence in a £4.2 billion reserve estimate is not a triumph of technology. It is a very expensive way of being wrong with conviction.

Preston is delighted. The actuary is not.

Preston will be back. He has a lot of board meetings to attend and a lot of things to enthusiastically misunderstand.

The Politics Actuary is a weekly cartoon series about actuaries, AI, the profession, and the absurdities of overregulation. New characters and themes are introduced as the series develops.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption | #7 Excel Relapse | #8 Holiday Risk Assessment

That should trouble anyone who reads accounts for a living. And yet these are the most comprehensive financial statements we have for the entire UK public sector. They're the closest thing to a corporate annual report for the country.

I used AI tools to extract and cross-reference data from the 2023-24 WGA, OBR forecasts, and HMRC statistics, then verified the key figures against the original sources. Here's what I think any actuary — or any numerate taxpayer — would want to know. This is the first in a series applying actuarial and accounting thinking to public finances. No politics. Just the numbers.

The COVID shadow

It's impossible to read these accounts without understanding what happened in 2020-21. The government's pandemic response — furlough, business support loans, NHS surge capacity, Universal Credit uplift, vaccine procurement — was the largest peacetime fiscal intervention in UK history. Borrowing hit £299bn in a single year (14.8% of GDP). Net debt jumped from around 80% of GDP to over 95%, where it remains. The WGA balance sheet swelled: total liabilities peaked at £6.3 trillion in 2021-22, inflated by pandemic borrowing and ultra-low discount rates on pension obligations.

The 2023-24 WGA captures the aftermath. The emergency spending has ended, but the structural consequences — higher debt, higher interest costs, expanded welfare claims — are now embedded in the baseline. The numbers you're about to read are post-COVID numbers. They are the new normal.

Revenue: three taxes fund two-thirds of the state

Total public services revenue in the year to 31 March 2024 was £1,019.9bn. Of that, £888.8bn was taxation and £131.1bn was other revenue (fees, interest income, NHS charges, rents, and so on).

Here's how the tax breaks down:

The top three taxes — Income Tax, VAT, and NICs — brought in £609.3bn. That's 69% of all taxation from just three instruments. Everything else — corporation tax, council tax, business rates, stamp duty, CGT, fuel duty, excise — combined produces just 31%.

This concentration matters. It means the Treasury is heavily exposed to the labour market. If employment falls or wage growth stalls, the majority of revenue is directly hit. The OBR's ready reckoners put this in concrete terms: a 1% fall in average earnings would reduce income tax receipts by £3.5bn and NICs by £2.8bn — a combined £6.3bn hit, roughly the entire annual inheritance tax yield, wiped out by a single percentage point of wage weakness.

Central government collects 94% of all taxation (£834.0bn). Local government collects just 6% (£54.8bn) — council tax and business rates. The centralisation of UK tax collection is extreme by international standards.

The people behind the numbers

Before looking at where the money goes, it's worth understanding who's doing what. The UK population (mid-2024, ONS: 69.3 million) breaks down roughly as follows:

Of the 43 million working-age adults, 34.2 million are in employment (2025) and 9.4 million are economically inactive — not in work and not looking for work. Against the working population, 13.1 million people receive the state pension. In total, 24 million people claim at least one DWP benefit, though many of those are also employed (Universal Credit, for example, tops up low wages).

The ONS breaks down the reasons for working-age inactivity, and the time series tells a striking story:

Long-term sickness has risen from 2.0 million to 2.8 million since 2019 — a 37% increase. It is now the single largest reason for working-age economic inactivity, overtaking students. This is the biggest structural shift in the post-COVID labour market, and it feeds directly into disability benefit costs (projected to rise from £39bn to £58bn by 2028-29).

Two other trends are worth noting. The number of women inactive due to looking after family has nearly halved since 1993 (from 2.9 million to 1.4 million) — a massive structural shift as more women entered the workforce. And early retirement among the working-age population peaked at 1.56 million in 2011 but has since fallen to 1.03 million, driven by the increase in state pension age from 60 to 66 for women and 65 to 66 for men.

The dependency arithmetic is simple: 13.1 million state pensioners supported by 34.2 million workers gives a ratio of roughly one pensioner for every 2.6 workers. That ratio will worsen as the baby boomer cohort retires and working-age population growth slows.

The migration variable

Net migration is the single biggest swing factor in this picture. It peaked at 944,000 in the year to March 2023 — driven by post-COVID catch-up, Ukraine and Hong Kong schemes, and a surge in health and care worker visas. It has since fallen sharply to 204,000 in the year to June 2025, back to roughly pre-pandemic levels.

The OBR's fiscal forecasts are highly sensitive to these numbers. In its March 2026 outlook, the OBR estimates that each year's net migrants contribute around £19,500 per person in tax revenue. Over its five-year forecast, the central migration assumption (settling at around 315,000 per year) adds £6.2bn to annual tax receipts by 2028-29 and reduces borrowing by £7.4bn. A swing of 200,000 per year in either direction would change borrowing by £13-20bn — comparable to the entire annual budget of some government departments.

The fiscal impact depends heavily on who is migrating. The OBR's average figure of £19,500 per migrant in tax revenue reflects a mix of high-earning professionals, students, care workers, dependants, and refugees — groups with very different fiscal profiles. A migrant arriving at age 25 and earning UK average wages until retirement would contribute a net £341,000 over their lifetime, according to OBR modelling. But that is an idealised scenario. Research from the Migration Observatory at Oxford shows that EEA migrants have historically made a positive net fiscal contribution, while non-EEA migrants — who now dominate UK immigration flows — tend to have a smaller or negative fiscal impact on average, largely because of lower average earnings and higher use of public services including education for dependant children.

The post-2020 surge was heavily non-EEA: students, health and care workers (many on relatively low salaries), dependants, and asylum seekers. Two specific humanitarian schemes account for a significant share: around 210,000 Ukrainians arrived under the Ukraine visa schemes by mid-2024, and approximately 163,000 Hong Kongers arrived under the BN(O) visa route by March 2025. These are distinct from the broader asylum system, where not all claims are upheld — the initial grant rate for asylum claims was 67% in 2023 but fell to 47% in 2024 under the higher standard of proof required by the Nationality and Borders Act, meaning that roughly half of claimants are not granted protection. None of the ONS net migration figures above include illegal immigration. The most visible route — small boat Channel crossings — totalled around 29,000 in 2023 and 37,000 in 2024, but this is only part of the picture. Others enter clandestinely via lorries and trains, or arrive legally and overstay their visas. The government does not publish an estimate of the total unauthorised population. Academic estimates from 2017 — before the small boat crossings proliferated — put it at 700,000 to 1.2 million (Pew Research Centre). Since then, around 175,000 unauthorised arrivals were recorded between 2020 and September 2024 alone, and roughly 85,000 refused asylum seekers from 2010-2023 have not been recorded as having left the UK. A reasonable current estimate of the total unauthorised population is probably somewhere above one million, though the true figure is inherently uncertain. This composition is likely less fiscally positive than the OBR's blended average implies. And the fiscal benefit of any cohort erodes over time as migrants settle, bring family, and eventually retire — at which point they draw on the same pensions and health services as everyone else. If public services spending adjusts to reflect a larger population (more school places, GP appointments, housing), the OBR estimates this would cost an additional £6-8bn — partially offsetting the tax gains.

The political direction is towards lower migration — higher salary thresholds, tighter visa rules, and a stated aim to reduce dependence on overseas workers. For the fiscal arithmetic, this creates a genuine tension: fewer working-age taxpayers entering the system at precisely the point when the baby boomer retirement wave is accelerating, but also potentially lower pressure on public services and housing. The net effect depends on whether the migrants who do come are higher-earning (as the salary threshold policy intends) or whether total numbers fall faster than average quality rises.

Expenditure: the two elephants in the room

Total expenditure on public services in the year to 31 March 2024 was £1,076.3bn, giving a net operating deficit of £56.4bn. Here's where it went:

Social security benefits (£311.4bn) deserve a closer look, because this is where the real money is:

*Universal Credit spiked to £80.4bn in 2020-21 at the height of the pandemic (temporary uplift + mass unemployment claims) before falling back and then rising again as legacy benefits migrate onto UC. The current £53.1bn reflects UC's growing share — 63% of combined UC and legacy benefit spending — as legacy benefits are wound down. By 2028-29, the OBR forecasts UC alone will reach £88.1bn.

The state pension alone — £127.3bn — is larger than the entire defence budget. Pensioner benefits collectively (including pension credit) total £133.0bn, or 43% of all social security spending. Working-age and disability benefits account for the other 57%.

The purchase of goods and services figure (£263.7bn) is worth unpacking, because at first glance it's surprising that it exceeds staff costs. This is the government's total non-staff operational spending — drugs and medical supplies, IT services, outsourced clinical and support services, equipment, maintenance, consumables, rent, utilities, and contracted-out services. The WGA Note 8 breakdown by department shows where it goes:

Health dominates — the DHSC spent £93.3bn on procurement alone, more than Defence, Scottish Government, and academies combined. This reflects the NHS's enormous outsourcing footprint: agency staff (classified as procurement, not staff costs), pharmaceutical purchases, PFI service charges, and contracted clinical services.

When you combine the functional spending view (from HM Treasury's COFOG classification), the picture is even starker:

Social protection and health together account for 53% of all government spending. These are the two elephants. Everything else the government does — defence, policing, courts, transport, housing, environment, culture — fits in the remaining 47%.

The balance sheet: £5 trillion of liabilities

The WGA consolidates a full accruals-based balance sheet as at 31 March 2024 — something the headline fiscal statistics (debt-to-GDP, borrowing) don't capture. This is where it gets interesting for actuaries.

Assets: £2,651.3bn (31 March 2024)

PP&E dominates — £1.5 trillion of roads, hospitals, schools, military equipment, social housing. Most of it is illiquid and non-tradeable. The government couldn't sell the M1 to plug a budget gap.

The gold holdings (£17.5bn) are a footnote — a reminder of Gordon Brown's sale of 395 tonnes at prices between $256 and $296 per ounce. At March 2024 prices (~$2,200/oz), the remaining 310 tonnes are worth roughly £17.5bn. The 395 tonnes sold would today be worth approximately £22bn.

Liabilities: £5,024.5bn (31 March 2024)

Net liabilities: £2,373.2bn — roughly £34,900 per person in the UK.

Government borrowings (£2,020.0bn) and pensions (£1,311.9bn) together are £3,331.9bn — two-thirds of all liabilities. The borrowings comprise gilts (£1,703.4bn carrying amount, split between £1,613.0bn non-current and £90.4bn current), NS&I products (£230.5bn), and Treasury bills (£86.1bn).

An accountant would note the fair value gap on the gilts: fair value is £1,463.1bn versus the carrying amount (book value) of £1,703.4bn. Fair value here is essentially market value — what the gilts would trade for in the open market given current yields. The book value is the amortised cost — roughly the face value the government must ultimately repay. The £240bn difference reflects above-market coupon rates on older issuances — the government is paying more interest than it would on today's borrowings, but the face value obligation remains.

Provisions: £262.2bn

The provisions note is where the long-tail liabilities live — and this is directly actuarial territory:

Nuclear decommissioning (£106.9bn) is one of the longest-tail liabilities in existence — stretching decades into the future, with cost estimates sensitive to discount rates, technology assumptions, and real cost inflation. Clinical negligence (£61.1bn across the UK) is growing rapidly and represents the NHS's hidden balance sheet problem.

The PFI legacy

Then there's PFI — the Private Finance Initiative. Between the 1990s and 2018, the government used PFI to build hospitals, schools, prisons, and roads using private sector capital. The idea was simple: the private sector builds and maintains the asset, the government pays an annual "unitary charge" over 25-30 years, and the risk sits with the contractor. The programme was discontinued in 2018, but 665 contracts are still running, with the last ones not expiring until 2048.

The WGA puts these on the balance sheet (Note 28). The net book value of PFI assets is £68.9bn. The corresponding liability — the present value of future capital repayments — is £35.9bn. But that's only the capital element. The total future commitments break down as follows:

HM Treasury's own 2024 summary data puts it slightly differently: £136bn in remaining unitary charge payments across the 665 projects, against a capital value of around £50bn.

The NAO's 2018 analysis was blunter. It found that PFI financing was around 40% more expensive than government borrowing for the same assets. Across the entire PFI portfolio, total lifetime repayments were estimated at roughly £300bn — for assets with a capital value of £57bn. That's a multiplier of over 5x.

The NHS is the most exposed. The Department of Health and Social Care holds £17.3bn of PFI finance obligations and £20.2bn of future service charges. There are 121 hospital PFI projects in England alone, with a combined capital cost of £11.6bn and estimated total repayments of £79.3bn. Individual cases are striking: Barts Health NHS Trust in London has a 43-year PFI contract for buildings worth approximately £1.1bn, with total repayments exceeding £7bn.

What makes PFI particularly interesting from an accounting perspective is the gap between how it appears in different sets of government accounts. In the WGA — which follows IFRS — most PFI assets and liabilities are on the balance sheet, because the government is deemed to control the asset. In the National Accounts — which politicians and the media cite — most PFI is off the balance sheet, because the private sector bears the risks and rewards. The WGA reconciliation shows £31bn of PFI contracts that appear in WGA but not in the headline public sector net financial liabilities. When politicians talk about "the national debt," they're using the number that excludes most of this.

140 of the 665 projects expire before 2030. The Infrastructure and Projects Authority recommends starting to plan for contract expiry at least seven years in advance. Many are already inside that window.

The pension iceberg

This is the section I find most interesting as an actuary. The WGA breaks down public sector pension liabilities by scheme (as at 31 March 2024), and the numbers are enormous:

These are almost entirely unfunded defined benefit occupational pensions — entirely separate from the state pension (which costs £127.3bn and appears in the social security benefits line above). There are no invested assets backing these schemes. Current employees and employers make contributions, current pensioners receive payments, and the gap is plugged by taxation. In 2023-24, total contributions to these unfunded schemes were £49.9bn and payments to pensioners were £55.0bn — a £5.1bn annual shortfall that taxpayers cover. (The WGA is published annually, typically 15 months after the year-end. The 2024-25 figures — as at 31 March 2025 — would be expected around mid-to-late 2026.)

The four largest schemes — NHS, Teachers, Civil Service, and Armed Forces — account for £1,161.3bn, or 88% of the total unfunded liability.

The discount rate illusion

Anyone who has worked with IAS 19 disclosures will immediately recognise what happened here. In my previous analysis of 12 FTSE companies' pension disclosures, the same dynamic played out at corporate level — but the government version is 100x larger.

Between 2022 and 2024, pension liabilities fell by £1,327bn — more than half. But the underlying pension promises are identical. No pensions were cut. No retirement ages were raised. No benefits were reduced. The only thing that changed was the discount rate used to value them.

A 3.6 percentage point rise in gilt yields halved the present value of future pension payments. If yields were to fall back to 2022 levels — which is not impossible in a recession — the liabilities would balloon back towards £2.6 trillion.

This is the single most important thing an actuary can explain about public finances: the numbers that politicians and commentators cite are extraordinarily sensitive to a single market variable that nobody controls and nobody can predict.

For unfunded government pensions, the real cash cost each year is what matters — contributions in versus pensions out. The present value is an accounting number. But that accounting number is what drives the WGA balance sheet, and it's what the headlines report. So when the Treasury points to "improving" net liabilities, it's worth asking: did anything actually change, or did the discount rate do the work?

The trend: 15 years of WGA data

The WGA has been published annually since 2009-10, giving us a 15-year picture:

Total assets have more than doubled (£1,208bn to £2,651bn). But liabilities have also more than doubled (£2,419bn to £5,025bn). The gap has roughly doubled too — from ~£1.2 trillion to ~£2.4 trillion.

The 2021-22 peak of £3.9 trillion in net liabilities was the combined effect of two forces: pandemic-era borrowing (which pushed government debt sharply higher) and ultra-low interest rates (which inflated pension liabilities to £2.6 trillion at a discount rate of ~1.5%). The subsequent "improvement" to £2.4 trillion was driven almost entirely by the discount rate rise — not by any reduction in spending commitments. Strip out the pension remeasurement, and the underlying fiscal position deteriorated: government borrowings rose from £1,754bn to £2,020bn between 2022-23 and 2023-24 alone, as the pandemic-era debt was rolled over at higher interest rates.

What's coming next

The OBR's March 2026 Economic and Fiscal Outlook projects borrowing of £133bn in 2025-26, falling to £59bn by 2030-31. Net debt is expected to peak at 96.5% of GDP in 2028-29 before stabilising.

But some of the spending pressures are baked in. The state pension triple lock guarantees minimum annual increases of 2.5%, earnings growth, or CPI — whichever is highest. State pension spending is already £127.3bn and growing faster than revenue. Disability benefits are projected to rise from £39bn to £58bn in five years. Debt interest — £63.4bn in the WGA, but forecast at £114bn for 2025-26 by the OBR — is now larger than the defence budget and growing. Much of this is the direct cost of pandemic borrowing: gilts issued at low yields in 2020-21 are being refinanced at rates two to three times higher, and index-linked gilts — roughly a quarter of the stock — amplified the cost when inflation surged in 2022-23.

Meanwhile, the personal allowance has been frozen at £12,570 since 2021-22 and won't rise until at least April 2031. A decade of fiscal drag, pulling millions of low earners into the tax base and pushing middle earners into higher bands. It's a stealth tax that compounds silently — and it's the government's main mechanism for closing the gap.

The tax-to-GDP ratio is forecast to reach 38% by 2030-31 — a post-war high.

If this were a household

Economists will tell you that a government is not a household. A household can't raise taxes, print its own currency, or live forever. All true. But the household analogy is useful as a stress test — not because the government is a household, but because the traditional reasons why it doesn't have to behave like one are becoming less convincing.

Let's scale it down. The UK government's finances in 2023-24, expressed as a household earning £50,000 a year:

In terms a mortgage adviser would understand:

• The mortgage (gilts — £99,000): A mortgage of roughly 2x income — modest by UK standards, where many households borrow 4-5x income. On its own, this would be very manageable. But here's what makes it unusual: the mortgage is less than half of this household's total liabilities. Most households have a mortgage and not much else. This one has a mortgage and a pension gap, and an overdraft, and repair bills, and a hire purchase contract. And it's adding to the mortgage every year — borrowing another £2,770 annually just to cover day-to-day expenses, which is like putting the weekly shop on the mortgage.

• The pension gap (unfunded schemes — £64,000): This household has promised itself a defined benefit pension but has put nothing aside to pay for it. It's as if you'd committed to paying yourself £2,700 a year in retirement, relying entirely on your children to cover it when the time comes. And the time is already here — the payments have started and contributions don't quite cover them.

• The other debts (other financial liabilities, provisions, payables — £83,000): This is a mix. The largest chunk (£59,000) is like an enormous overdraft facility that the household arranged during a crisis and hasn't unwound — the equivalent of the Bank of England's QE reserves sitting on the balance sheet. Buried within that overdraft is a hire purchase agreement (PFI: £1,800 on the balance sheet, but £5,600 in total future payments) where you're paying five times the asset's value over 25 years and can't get out of the contract. Then there's £13,000 of known future repair bills — the roof needs replacing (nuclear decommissioning: £107bn), there's a growing legal claim from a neighbour (clinical negligence: £61bn), and various other commitments that can't be avoided. And £11,000 in unpaid bills sitting on the kitchen table.

• The interest bill (£3,110 a year): That's 6.2% of income going on servicing debt — more than half what the household spends on its children's education (£5,800). And it's rising — the old fixed-rate deals are expiring and being replaced at rates two to three times higher. The OBR forecasts this rising to over £5,600 by 2025-26.

• The house (assets — £130,000): Mostly property, roads, and equipment. Theoretically worth £130,000, but almost none of it can be sold. You can't liquidate a hospital or auction off the M1 to pay down debt.

Add it all up and this household has £246,000 in liabilities against £130,000 in unsellable assets, is spending more than it earns every month, and has no realistic plan to pay down the balance. A financial adviser would call this unsustainable. The traditional response is that governments have three escape routes that households don't:

1. Raise taxes. But the tax-to-GDP ratio is already heading for a post-war high of 38%. The personal allowance has been frozen for a decade. National Insurance was raised in 2022 and employer NICs again in 2025. At some point, higher taxes reduce economic activity and the revenue gains diminish — the Laffer curve is real even if nobody agrees where the peak is. Capital and talent are more mobile than ever: raise corporation tax too far and investment moves; raise personal tax too far and high earners relocate. The headroom is shrinking.

2. Print money (monetary financing). The Bank of England effectively did this during COVID through quantitative easing — buying £450bn of gilts with newly created reserves. The 2022-23 inflation surge was primarily driven by energy prices (Russia's invasion of Ukraine sent gas prices soaring) and global supply chain disruptions, but QE created the inflationary backdrop: an economy flooded with liquidity was far more susceptible to price shocks when they arrived. The combined result — CPI hitting 11.1% in October 2022 — eroded real wages, inflated index-linked gilt costs, and forced emergency rate rises. The lesson is fresh: monetary financing may not directly cause inflation, but it removes the buffer that would have contained it. Whether that lesson prevents it being used again is another matter — governments under fiscal pressure have historically found reasons to revisit tools they once swore off.

3. Grow your way out. This is the only genuinely sustainable route — if real GDP growth exceeds the real interest rate on government debt, the debt-to-GDP ratio falls naturally over time. But the OBR's central forecast is for real growth of 1.1-1.6% per year through 2030. Gilt yields are around 4.5%. The arithmetic only works if growth accelerates substantially or yields fall back — neither of which is guaranteed.

There is a fourth factor that rarely features in these discussions: AI and automation. If AI delivers even a fraction of its projected productivity gains, it could transform the fiscal equation — higher output per worker, new industries, expanded tax base. But it could equally hollow out the traditional employment model that generates 69% of all tax revenue (income tax + NICs + VAT on consumption). If millions of jobs are automated faster than new ones are created, the government faces lower tax receipts and higher welfare costs simultaneously. Nobody knows which effect will dominate, but the risk is asymmetric: the upside accrues gradually over decades, while the labour market disruption could arrive in years.

The honest assessment is this: the UK's fiscal position is not yet a crisis, but the margin of safety has narrowed dramatically. Government debt is 198% of annual revenue — nearly two years' income. Interest costs consumed 6.2% of revenue in 2023-24, and the OBR forecasts this rising to over 11% by 2025-26 as pandemic-era gilts are refinanced at higher rates. The major spending categories — pensions, health, disability — are all structurally growing faster than revenue. Every traditional fiscal lever has already been partially used. And the next recession — whenever it comes — will hit a balance sheet with far less capacity to absorb the shock than in 2008 or 2020.

A household in this position would be told to cut spending, increase income, or both — urgently. A government has more tools, but fewer than it used to, and the cost of each tool is higher than it was a decade ago.

Why this matters for actuaries

Actuaries value long-term liabilities for a living. We understand that a discount rate is not a magic wand — it changes the present value, not the cash flows. We know that unfunded promises are real obligations even when they don't appear on a conventional balance sheet. We're trained to stress-test assumptions and ask what happens when they're wrong.

The WGA is the single most actuarially relevant document the UK government publishes, and almost nobody reads it. Not politicians, not journalists, and — based on the NAO's audit disclaimer — not even all the public bodies that are supposed to contribute to it.

In the next post in this series, I'll go deeper into the pension liabilities — how the six unfunded schemes compare, what the cash flow projections look like, and what longevity risk means when there's no corporate sponsor and the bill falls directly on future taxpayers.

All figures in this post are from the Whole of Government Accounts 2023-24 (HC 917, published 17 July 2025), the OBR March 2026 Economic and Fiscal Outlook, and OBR tax-by-tax spend-by-spend briefings. The WGA received a disclaimed audit opinion from the Comptroller and Auditor General for the second consecutive year. Full source links are available on request.

Patrick Lee is the founder of INQA Group. He is a qualified actuary — MA FIA (1990 to Sep 2020) Dip Stats — and a software architect.

Most people book a holiday by scrolling through TripAdvisor and picking somewhere with a pool. An actuary builds a risk matrix, plots "lost passport" and "Lily's phone falls in pool" by likelihood and impact, and concludes that Greece has an unacceptable risk profile.

The family's complete lack of surprise is the real punchline. Beth has been through this before. She already knows they're going to Greece — she just has to wait for Patrick to exhaust his analysis first.

The Politics Actuary is a weekly cartoon series about actuaries, AI, the profession, and the absurdities of overregulation. It follows a dad who can't switch off his professional brain, a family who've learned to live with it, and an inexplicable pet crocodile.

Any resemblance to real actuaries is entirely intentional.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption | #7 Excel Relapse

Every firm has announced its AI strategy. Every team has someone who has "started using AI." And somewhere, in every office, someone is quietly doing the whole thing in Excel first and then asking AI if it agrees — just so they can tick the box.

The profession's attachment to spreadsheets is not a weakness. It is a survival instinct honed over decades of being told that the new tool will change everything, only to discover that it crashes on the edge case your Excel sheet handled in 2014.

The Politics Actuary is a weekly cartoon series about actuaries, AI, the profession, and the absurdities of overregulation. It follows a dad who can't switch off his professional brain, a family who've learned to live with it, and an inexplicable pet crocodile.

Any resemblance to real actuaries is entirely intentional.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone | #6 Long-Term Assumption

In my first post, I used AI to extract and compare IAS 19 pension disclosures from four FTSE companies — BT, BAE Systems, Tesco, and BP. That was a snapshot: one year, four companies, a proof of concept.

This time I've gone deeper. I expanded to 12 FTSE companies and built time series going back up to five years — capturing the full before-and-after of the 2022 gilt yield crisis. The headline numbers are dramatic: UK discount rates roughly tripled (from ~1.5% to ~5.5%), and defined benefit obligations shrank by tens of billions of pounds. But the story is far more nuanced than "rates up, liabilities down."

The key lies in a metric I've derived from public data that, as far as I can tell, nobody else is routinely computing from annual report disclosures: the hedge ratio.

The rate surge in context

UK pension discount rates went from ~1.5% (2020) to ~5.5% (2024/25) — roughly a 400 basis point increase over four years. The bulk of the move (~250–300bp) happened in a single year, 2022, accelerated by the September mini-budget.

For December 2024 year-ends, UK scheme discount rates now cluster in a remarkably tight band: 5.44% (Barclays) to 5.55% (Lloyds). March 2025 year-ends are slightly higher (5.70–5.75%) reflecting late-March gilt yields. The profession is effectively pricing off the same curve.

The impact on liabilities was enormous. DBOs fell 6–16% in the most recent year alone across all 12 companies. Across my sample, total DBOs shrank from roughly £267bn to £237bn — a ~£30bn reduction in the most recent year alone, on top of much larger falls in 2022.

So far, so straightforward. But the net position — the number that actually matters — tells a very different story depending on which company you look at.

The hedge ratio: a single number that explains everything

Every IAS 19 disclosure includes OCI (Other Comprehensive Income) remeasurements. These break out how much of the change in assets and liabilities came from financial assumption changes — primarily interest rates. By comparing asset sensitivity to liability sensitivity, you can compute how well a scheme's investment portfolio tracks its liabilities when rates move.

I call this the hedge ratio: asset loss from rate movements divided by liability gain from rate movements, expressed as a percentage. A hedge ratio of 100% means perfect tracking — if rates rise and liabilities fall by £1bn, assets also fall by £1bn, and the net position is unchanged. Below 100% means the assets aren't fully tracking liabilities (under-hedged). Above 100% means the assets have more rate sensitivity than the liabilities — typically because of leveraged LDI.

This is derivable from public OCI data. You don't need access to scheme accounts or investment reports. Yet it's arguably the single most revealing metric for understanding pension risk.

Here's what it shows for my original four companies:

Four companies, four hedge ratios, four completely different outcomes

BP — hedge ratio 98%: the benchmark

Despite a 41% collapse in DBO ($49.9bn to $29.5bn in a single year), BP's surplus barely changed — $4.1bn to $4.0bn. Assets and liabilities fell in near-perfect lockstep. A 98% hedge ratio means almost all rate sensitivity was matched. This is the benchmark for how LDI should work. When the hedge ratio is right, even extreme market moves pass through harmlessly.

BT — hedge ratio 121%: over-hedged

BT's DBO fell 38% over four years — from £59.7bn to £36.9bn. Yet the deficit only improved by 20% (from £5.1bn to £4.1bn). Why?

BT's leveraged LDI portfolio had more rate sensitivity than the liabilities it was hedging. In the year to March 2023, asset losses from rate movements (£14.9bn) exceeded liability gains (£12.3bn) by £2.6bn. Rate rises, which should have helped, actually widened the deficit.

BT's pension story is one of volatility, not resolution — a rollercoaster driven by over-hedging.

Tesco — hedge ratio 124%: also over-hedged

The most counterintuitive result. Tesco went from a £2.8bn surplus to a £0.4bn deficit in a single year, despite discount rates nearly doubling (2.8% to 4.9%). With a 124% hedge ratio, every 1% rate rise reduced liabilities by X but reduced assets by 1.24X — destroying value on a net basis. Over-hedging via leveraged LDI means rate rises hurt rather than help.

BAE Systems — hedge ratio 53%: under-hedged

The most dramatic swing — from £2.3bn deficit to £1.5bn surplus in one year. But this wasn't skilful management — it was an under-hedged position that got lucky. When rates surged, group liabilities fell by £11.8bn but assets only fell by £6.3bn. BAE's asset portfolio simply wasn't tracking its liabilities.

If rates had fallen by the same amount, the deficit would have roughly tripled to ~£7.8bn. What looks like a turnaround was actually an unhedged bet that happened to pay off.

The full picture: all 12 hedge ratios

I computed the same metric for all 12 companies. The results split cleanly into three groups:

Hedge ratios computed from OCI remeasurement disclosures: |asset return excl. interest income| / |actuarial gain from financial assumptions on DBO|. Figures based on each company's most recent annual report (2024 or 2025 year-end). Note: these ratios reflect current investment strategies; some companies (notably BT and Tesco) were significantly more hedged in 2022 when the large rate moves occurred — see narrative sections below.

The pattern is striking. Five companies show hedge ratios above 100%, with Lloyds the most extreme at 136%. At the other end, seven companies are under-hedged — their pension positions are effectively making unhedged bets on the direction of interest rates.

What's particularly interesting is that the picture has shifted since 2022. BT and Tesco, which were over-hedged during the gilt crisis (121% and 124% respectively in the year of the big rate move), have since de-leveraged to 92% and 56%. BP has moved the other way — from 98% in 2022 to 127% now. Investment strategies are not static.

Only three companies — Rolls-Royce, Vodafone, and BAE — sit in the 89–103% zone where rate movements pass through without materially affecting the net position.

The leveraged LDI cluster

The companies with hedge ratios above 120% — Lloyds, BP, and Barclays — all share the hallmarks of leveraged LDI: their assets have more interest rate sensitivity than their liabilities.

Lloyds at 136% is the most extreme case in my sample. In 2024, rising rates reduced Lloyds' pension obligations by £2.7bn — but asset losses were £3.7bn. The over-hedging cost £1.0bn on the financial assumptions line alone. Other factors (contributions, experience gains) clawed back some of that, but the surplus still fell from £3.5bn to £2.9bn despite everything moving in what should have been the "right" direction.

The mechanism is visible in Lloyds' asset allocation: 75% of scheme assets are in government bonds (23% fixed-interest, 52% index-linked). The fixed-interest gilts hedge nominal rate movements; the index-linked gilts hedge real rates and inflation. But the combined portfolio has more rate sensitivity than the liabilities it's matching — the hallmark of leveraged LDI, where schemes use repo and swap overlays to amplify their hedging beyond 100%.

BP at 127% is a new entrant to the over-hedged club. In the 2022 rate surge, BP's hedge ratio was 98% — near-perfect. But its most recent OCI shows asset losses of $2.5bn against liability gains of $2.0bn. The UK plan alone shows an even more extreme 160% ratio ($2.4bn asset loss vs $1.5bn liability gain). BP appears to have increased its LDI leverage, particularly through its extensive gilt repo programme ($5.7bn of repo debt funding index-linked and nominal gilts).

Barclays at 122% tells a similar story — asset losses of £2.2bn against liability gains of £1.8bn. Like Lloyds, Barclays can afford the drag because of its substantial surplus (£3.0bn). But the over-hedging cost approximately £400m in the year.

The banking sector's over-hedging is no accident. UK bank pension schemes have historically pursued aggressive LDI strategies, often using gilt repo, interest rate swaps, and inflation swaps to amplify their liability-matching exposure beyond 100%. The logic was that in a falling-rate environment, over-hedging provides extra protection. But rates didn't fall — they tripled. And the leverage worked in reverse.

BT and Tesco were also over-hedged in 2022 (121% and 124% respectively) — but have since de-leveraged significantly. BT's most recent hedge ratio is 92% and Tesco's is 56%. The 2022 gilt crisis may have prompted a strategic rethink. This is an important observation: hedge ratios are not fixed. Investment strategies evolve, and a company that was over-hedged during the crisis may have a very different risk profile today.

Rolls-Royce provides an interesting contrast. Its OCI-derived hedge ratio is 103% — almost perfect. But its sensitivity disclosure tells a different story: a 0.25% decrease in discount rates would increase obligations by £145m while increasing LDI assets by £179m. That's a forward-looking hedge ratio of 123%. This suggests Rolls-Royce may have increased its LDI leverage during 2024, moving from a well-hedged to an over-hedged position — something to watch in future disclosures.

The under-hedged: lucky, not skilful

At the other end, Shell (52%), Tesco (56%), GSK (61%), AstraZeneca (72%), BAE (89%), and BT (92%) are all currently under-hedged — their assets have less rate sensitivity than their liabilities. When rates rose, this worked in their favour: assets didn't fall as much as liabilities, so net positions improved.

But this cuts both ways. If rates reverse, these schemes would be the most exposed. An under-hedged scheme that benefits from rising rates is not demonstrating good risk management — it's demonstrating that an uncontrolled exposure happened to move in a favourable direction.

Shell's 52% hedge ratio is particularly notable given its size ($66bn DBO, the largest in my sample). When rates rose, Shell's liabilities fell by $4.4bn from financial assumption changes, but assets only fell by $2.3bn — a $2.1bn windfall. Shell's surplus grew from $1.6bn to $3.3bn. But with only half the rate sensitivity matched, a rate reversal would be painful.

The bigger picture: de-risking and the end of DB

Stepping back from hedge ratios, the broader picture across all 12 companies is one of structural transformation:

Seven companies now show net pension surpluses — Shell, BAE, BP, Lloyds, Barclays, Vodafone, and National Grid. Three years ago, the majority were in deficit.

Only BT and AstraZeneca have material deficits — BT at £4.1bn and AstraZeneca at $1.2bn. Everyone else is either in surplus or near balance (GSK at 99% funded, Tesco and Rolls-Royce just below 100%).

De-risking has reached the end-game. Shell executed a $5.1bn pension buy-out in 2024. National Grid has 31% of its UK assets in annuity policies. GSK has a £340m buy-in covering one plan. AstraZeneca has a longevity swap covering $2.2bn of liabilities. Barclays expects zero employer contributions in 2025. The direction is clear — UK defined benefit pensions are being systematically wound down via insurance market transfer.

DC has won. Where both costs are disclosed, defined contribution pension costs now exceed defined benefit costs at every single company in my sample. Lloyds: £537m DC vs a net DB credit. Shell: $514m DC vs $295m DB. The structural shift to DC is complete — DB is purely a legacy obligation being run off.

The current net position across all 12 companies

What actuaries should know

The hedge ratio — derivable from public OCI data in any IAS 19 disclosure — is arguably the single most revealing metric for understanding pension risk. It explains why BP sailed through the gilt crisis while BT and Tesco didn't. It reframes BAE's apparent "turnaround" as an unhedged bet. It reveals that Lloyds' £27bn pension scheme has more rate sensitivity in its assets than in its liabilities. And it's not something I've seen routinely computed from annual report data.

Five of 12 companies currently show hedge ratios above 100%, consistent with leveraged LDI. But the picture is evolving. BT and Tesco, which were significantly over-hedged during the 2022 gilt crisis, have since de-leveraged — while BP has moved from well-hedged to over-hedged. The September 2022 crisis exposed the risks of leveraged LDI when it triggered collateral calls across the UK pension sector, and some schemes clearly responded by reducing their leverage. But the OCI data shows that over-hedging has not disappeared — it has simply migrated.

The disclosure gap around leverage is a problem. The hedge ratio I'm computing from OCI data is effectively a backdoor into information that should be disclosed directly but isn't. Some companies are transparent: BP explicitly shows $5.7bn of repo debt in its asset table, and Tesco breaks out £3.3bn in repo agreements and swaps within its LDI portfolio. But Lloyds buries £(8.1)bn in a single "money market, cash, derivatives, other" line with no decomposition. Barclays shows derivatives of £(1.8)bn without further detail. There is no IAS 19 requirement to disclose the degree of leverage or the instruments creating it.

This matters because leverage fundamentally changes the risk character of a pension scheme. When I was an investment partner at Watson Wyatt in 1994 to 1995 — and their in-house expert on derivatives — I used to advise pension plan trustees against any form of gearing. My view then, and now, is that pension plan assets should never be in a position where they could become negative. Under leverage, that is theoretically possible: if markets move far enough against the leveraged position, the borrowed obligations can exceed the asset value. A pension scheme is supposed to be a pool of assets backing promises to retirees. Leverage turns it into something closer to a hedge fund.

The OCI-derived hedge ratio makes this visible from public data. But it shouldn't require reverse-engineering. The degree of leverage — gross and net exposure, by instrument type — ought to be a standard IAS 19 disclosure.

This kind of cross-company, multi-year analysis is exactly the sort of work that used to take weeks. I built this dataset — 12 companies, up to five years of data, full extraction of assumptions, balance sheets, income statements, contributions, asset allocations, sensitivities, and membership — in a few hours using AI-assisted extraction from publicly available PDFs. The extraction prompt is reusable: point it at any company's annual report and it pulls out the same structured dataset.

The data behind this analysis is available as a premium download for INQA members — the full 12-company time series dataset, individual company extraction files, and the reusable extraction prompt template.

What's next

I'm planning to extend the time series to five years for all 12 companies (currently I have five years for BT, BAE, and BP; four for Tesco; two to three for the rest). I'd also like to add more companies and build an automated extraction pipeline.

If there's a specific company or analysis you'd like to see, let me know.

And if you haven't taken INQA's AI skills baseline assessment yet, it takes about 5 minutes and gives you a personalised score: Take the assessment

All data extracted from publicly available annual reports (2020–2025). No proprietary data was used. Analysis assisted by Claude (Anthropic). Hedge ratios computed from OCI remeasurement disclosures.

Every actuary has sat in a room where someone wrote a number on a whiteboard, circled it with great authority, and called it a long-term assumption. The fact that the world outside the window looked nothing like the number was, apparently, beside the point.

The profession's ability to define away inconvenient facts with a straight face is genuinely impressive. It is also, occasionally, worth laughing at.

The Politics Actuary is a weekly cartoon series about actuaries, AI, the profession, and the absurdities of overregulation. It follows a dad who can't switch off his professional brain, a family who've learned to live with it, and an inexplicable pet crocodile.

Any resemblance to real actuaries is entirely intentional.

Previous cartoons: #1 Record Investment | #2 Y-Axis Trick | #3 Prompt Engineering | #4 Transparency Request | #5 Define Everyone

I took four FTSE companies — BT Group, BAE Systems, Tesco, and BP — downloaded their latest annual reports, and used AI to extract and compare their IAS 19 pension disclosures. The whole process, from downloading the first PDF to having a completed comparison table, took about an hour.

Here's what I found — and what it tells us about using AI for actuarial analysis.

The process

The pipeline was straightforward:

1. Download annual reports — financial statements PDFs from each company's investor relations page (BT, Tesco) or from companiesmarketcap.com (BAE, whose site blocks automated downloads)
2. Extract pension disclosure data — using a standardised prompt template that asks for 20+ data points across assumptions, balance sheet, income statement, contributions, asset allocation, sensitivities, and membership
3. Normalise and compare — bringing the four extractions into a single comparison table, adjusting for different year ends, age bases, sensitivity increments, and currencies

The extraction prompt template is reusable — point it at any UK company annual report and it pulls out the same structured dataset. That's the bit that makes this scalable.

The headline numbers

BP reports in USD ($1 ≈ £0.79 at 31 Dec 2024). Its UK plan assumptions (discount rate, inflation, mortality) are sterling-based and directly comparable to the other three companies — only the monetary amounts are translated.

Six things that stand out

1. BT is an outlier

BT's pension scheme dominates the picture — a £4.1bn deficit with annual cash contributions of £856m, scheduled to continue at £780m pa until 2030. The other three companies are in or near surplus. BT's cash outflow is dominated by deficit repair contributions (£803m of the £856m total) — a direct consequence of the shortfall. By contrast, Tesco's scheme is nearly fully funded and requires just £40m a year.

2. Discount rates converge

All four companies use discount rates in a tight 5.5%–5.75% band, consistent with the WTW FTSE 100 median of 5.55%. BT's slightly higher rate (5.75%) reflects its later year end — gilt yields rose during Q1 2025. Ironically, that timing works in BT's favour: had it used 5.5% like BAE and BP, its DBO would have been higher and the deficit even larger.

3. Everyone uses CMI 2023

All four companies have adopted CMI 2023 mortality projections (Tesco with a slightly unusual CMI 2020 base projected to 2018, then CMI 2023 for future improvements). Long-term improvement rates range from 1.0% pa (BAE, Tesco) to undisclosed (BT).

4. Life expectancy varies more than you might expect

Male life expectancy at 65 ranges from 19.6 years (Tesco) to approximately 22 years (BP's UK scheme). Female life expectancy shows a similar pattern — 22.2 years at Tesco versus approximately 24 years at BP. The gap is consistent across both sexes, which points to the socio-economic profile of each company's workforce rather than differences in sex mix. Tesco's largely blue-collar scheme population has materially lower longevity than BP's.

BT goes further — splitting life expectancy by "pension bracket" (higher vs lower earners within the same scheme), which explicitly models socio-economic mortality differentials.

5. DC now exceeds DB everywhere

All four companies spend more on defined contribution pensions than on their defined benefit schemes. BT: £305m DC vs £225m DB. Tesco: £454m DC vs £49m DB. Three of the four schemes (BT, Tesco, BP UK) are fully closed to accrual — reflecting past service only. BAE is the exception: its scheme is closed to new entrants but existing members are still accruing benefits, which is why its current service cost (£134m) is roughly 10x higher than BT's or Tesco's.

6. De-risking is universal

Growth asset allocations range from approximately 25% (BP) to 45% (Tesco). LDI strategies, longevity hedging, and cash flow matching are in place across all four schemes. BP has the most defensive allocation — 48% in index-linked government bonds alone.

A question worth asking: given the likely significant disruption to the economy from AI over the next decade — arguably earlier — are index-linked government bonds really going to remain low risk? What happens to sovereign creditworthiness when income tax receipts fall significantly? The de-risking consensus assumes a continuity that AI may not deliver.

Key assumptions comparison

BT and Tesco both disclose bifurcated RPI/CPI assumptions reflecting the 2030 RPI reform (when RPI will be aligned with CPIH, narrowing the RPI-CPI gap significantly) — a nuance that older analyses would miss.

What AI can and can't do here

What worked well:

• Extracting structured data from narrative pension notes — AI handles this faster and more consistently than manual reading
• Identifying which note contains the IAS 19 disclosures and pulling the right pages
• Producing structured markdown that's easy to compare across companies

What needed human judgement:

• Normalising across different disclosure formats (BAE quotes life expectancy as absolute age, not years remaining; BT presents sensitivities as charts, not tables; BP reports in USD)
• Spotting subtleties like Tesco's unusual CMI base year treatment
• Deciding which figures to use when companies have multiple schemes or report at different levels (UK only vs total group)

This isn't "AI replaces the actuary." It's "AI does the extraction in minutes instead of hours, so the actuary can focus on the analysis."

Why this matters for actuaries

Comparative IAS 19 analysis is a consulting product. Firms charge real money to produce benchmarking reports that look a lot like the table above. The data is all public — it's just time-consuming to extract manually.

With a reusable extraction prompt and an hour of time, I produced a comparison across four major companies. Scaling to 10 or 20 companies would take proportionally longer but the process is the same. Adding historical years to show trends over time is the obvious next step.

If you're an actuary who's been wondering what AI could actually do in your day-to-day work — this is a concrete example. Not hypothetical, not a toy demo. Real data, real companies, real findings.

What's next

I'm planning to expand this to more companies and add time series data. If there's a specific company or comparison you'd like to see, let me know.

And if you haven't taken INQA's AI skills baseline assessment yet, it takes about 5 minutes and gives you a personalised score: Take the assessment

All data extracted from publicly available annual reports (BT Group FY 2024/25, BAE Systems FY 2024, Tesco FY 2024/25, BP FY 2024). No proprietary data was used. Analysis assisted by Claude (Anthropic).

Set in a fictional household — a dad who can't switch off his professional brain, a family who've learned to live with it, and an inexplicable pet crocodile — the series takes on politics, the profession, everyday life, and the absurdities of overregulation.

Cartoon #5 in the series. When an eleven-year-old deploys the "everyone has one" argument at the dinner table, most parents cave. An actuarial parent reaches for the notepad.

Set in a fictional household — a dad who can't switch off his professional brain, a family who've learned to live with it, and an inexplicable pet crocodile — the series takes on politics, the profession, everyday life, and the absurdities of overregulation.

"She'd spent her whole career promoting transparent financial disclosure from pension funds. Her own professional body seemed to regard similar transparency as unnecessary despite having highly numerate members."

The profession that built its reputation on transparent financial analysis apparently regards similar transparency about its own finances as unnecessary.

Any resemblance to real actuaries is entirely intentional.

New cartoons every week. Follow on LinkedIn, pjlee.net or X (@politicsactuary).

Any resemblance to real actuaries is entirely intentional.

You've spent three hours teaching the computer to think exactly like you. You could have just done the work yourself." "That's not the point, Beth."

I've spent time pulling together this curated reference: over 107 freely accessible, regularly updated datasets relevant to actuarial work. It covers UK sources in depth (the ONS, Bank of England, Environment Agency, and many more) and extends internationally to the US, Canada, Australia, the EU, and global bodies like the WHO, World Bank, and UN.

This isn't a lightweight listicle. It's meant to be a working reference — something you bookmark and come back to when you need data for a specific project. Every link points to an established, official source. Every description is drawn from the actual dataset documentation.

A few things to note before diving in:

• All datasets listed here are free. Some require free registration (noted where applicable), but none require paid subscriptions.
• CMI data is not included. The Continuous Mortality Investigation mortality projection models (e.g. CMI_2023) are the industry standard for UK pension scheme mortality assumptions, but they require an IFoA subscription. The ONS data they're partly derived from is included below.
• Government URLs can change. If a link doesn't resolve, search the publisher's website for the dataset name — the dataset names and publishers are the stable identifiers.

Table of Contents

UK Datasets

1. Mortality and Life Tables
2. Population Projections
3. Pension Statistics
4. Road Traffic and Motor Insurance
5. Crime Statistics
6. Flood Risk and Climate
7. Property Data
8. Health and Disease
9. Income and Earnings
10. House Prices
11. Inflation and Price Indices
12. Interest Rates and Gilt Yields
13. Student Loans
14. Census and Population Estimates
15. Migration and Births

International Datasets

16. International Mortality and Life Tables
17. Insurance and Claims Data
18. International Health Data
19. International Demographics and Population
20. International Finance and Economics
21. Pensions and Social Security
22. Catastrophe and Climate Data
23. Additional Specialist Sources

Reference

24. Quick Reference by Actuarial Specialty
25. Multi-Domain Data Portals
26. Tips for Working with These Datasets

UK Datasets

1. Mortality and Life Tables

National Life Tables, United Kingdom — ONS Period life tables giving mortality rates (qx), survival probabilities, and life expectancy by single year of age and sex. Based on population estimates and death registrations. Separate tables for England and Wales, Scotland, and Northern Ireland. Updated annually. ons.gov.uk — National Life Tables

Single Year Life Tables, England and Wales — ONS Historical single-year life tables from 1841 onwards. Invaluable for long-term mortality trend analysis. ons.gov.uk — Single Year Life Tables

Death Registrations Summary Tables, England and Wales — ONS Annual deaths by age, sex, and cause. Includes crude and age-standardised mortality rates. Also the primary source for leading causes of death analysis by ICD-10 chapter. Updated annually. ons.gov.uk — Death Registrations

Weekly Death Registrations, England and Wales — ONS Weekly provisional death counts by age group. Widely used for monitoring excess mortality during COVID-19 and remains a key real-time mortality indicator. ons.gov.uk — Weekly Deaths

Mortality Statistics by Cause of Death — ONS / NOMIS Detailed mortality by cause of death, age, sex, and area. Based on ICD-10 coding. NOMIS provides flexible query tools for custom tabulations. nomisweb.co.uk

Death Statistics, Scotland — National Records of Scotland Scottish death registrations, life tables, and mortality rates by age, sex, cause, and area. Separate from ONS England and Wales data. Includes quarterly and weekly provisional figures. nrscotland.gov.uk — Deaths

Death Statistics, Northern Ireland — NISRA Northern Ireland death registrations and mortality statistics by age, sex, and cause. nisra.gov.uk — Deaths

2. Population Projections

National Population Projections — ONS Principal and variant projections by single year of age and sex for the UK. Includes high/low fertility, mortality, and migration variants. Essential for longevity and demographic assumptions in pension valuations. Updated every two years. ons.gov.uk — National Population Projections

Subnational Population Projections — ONS Population projections by local authority, age, and sex. Useful for regional pension and insurance analysis. ons.gov.uk — Subnational Projections

3. Pension Statistics

Occupational Pension Schemes Survey (OPSS) — ONS Members, contributions, assets, and benefits by scheme type (DB/DC). Tracks the ongoing shift from defined benefit to defined contribution. Updated annually. ons.gov.uk — OPSS

Scheme Return Data — The Pensions Regulator Aggregated data from pension scheme returns including membership, funding levels, and demographics. Includes the Purple Book (PPF/TPR annual publication on the UK DB pensions landscape). thepensionsregulator.gov.uk — Scheme Return Data

Pensioners' Incomes Series — DWP Detailed analysis of pensioner incomes by source, age group, and family type. Key for understanding retirement income adequacy. gov.uk — Pensioners' Incomes

State Pension Statistics — DWP (via Stat-Xplore) Interactive tool providing state pension recipient data by age, amount, and geographical area. Also covers other DWP benefits. Free registration required. stat-xplore.dwp.gov.uk

Family Resources Survey — DWP Major household survey covering income, pension participation, savings, disability, and housing costs. Sample of approximately 20,000 households per year. Microdata available via UK Data Service. gov.uk — Family Resources Survey

4. Road Traffic and Motor Insurance

STATS19 Road Safety Data — Department for Transport The definitive UK road accident dataset. Every injury road accident reported to police in Great Britain, with details of circumstances, vehicles, and casualties. Three linked tables: Accident, Vehicle, and Casualty. Georeferenced. data.gov.uk — Road Safety Data

Road Traffic Statistics — Department for Transport Traffic volume data (vehicle miles/kilometres) by road type, vehicle type, and region. Essential exposure denominators when analysing STATS19 data. gov.uk — Road Traffic Statistics

MOT Test Data — DfT / DVSA Anonymised MOT test results for all vehicles in Great Britain: make, model, year, mileage, test outcome, and failure reasons. Hundreds of millions of records. data.gov.uk — MOT Tests

5. Crime Statistics

Police Recorded Crime — Home Office / data.police.uk Street-level crime and outcome data for every police force in England, Wales, and Northern Ireland. Each record includes crime type, approximate location, and outcome. Millions of records per year. data.police.uk

Crime Survey for England and Wales (CSEW) — ONS Survey-based crime data including crimes not reported to police. Covers household crime, personal crime, and fraud/cybercrime, giving a broader picture than police-recorded data alone. ons.gov.uk — Crime Survey

6. Flood Risk and Climate

Environment Agency Flood Maps — Environment Agency Flood zone maps (Zones 1, 2, 3a, 3b), flood risk from rivers and sea, and surface water flood risk. Spatial data for England. Critical for property insurance pricing and exposure management. environment.data.gov.uk

Met Office HadUK-Grid — Met Office / CEDA Archive Gridded climate observations for the UK: temperature, rainfall, sunshine, wind, humidity. 1km resolution from 1836 to present. The definitive observational climate dataset for the UK. metoffice.gov.uk — HadUK-Grid

UK Climate Projections (UKCP) — Met Office Probabilistic climate projections for the UK under different emissions scenarios. Temperature, precipitation, sea level, wind. Essential for long-term climate risk assessment. metoffice.gov.uk — UKCP

SEPA Flood Maps, Scotland — Scottish Environment Protection Agency Flood risk maps for Scotland, covering river, coastal, and surface water flooding. sepa.org.uk — Flood Maps

7. Property Data

Price Paid Data — HM Land Registry Every residential property transaction in England and Wales since 1995. Includes price, date, postcode, property type (detached, semi, terraced, flat), new build flag, and tenure. Updated monthly. gov.uk — Price Paid Data

Energy Performance Certificate (EPC) Data — MHCLG EPC ratings for domestic and non-domestic buildings in England and Wales. Includes energy efficiency rating, floor area, wall type, heating type, and construction age. Over 20 million certificates. Free registration required. epc.opendatacommunities.org

8. Health and Disease

Hospital Episode Statistics (HES) Summary Data — NHS England Summary statistics on hospital admissions in England: admitted patient care, outpatient attendances, and A&E attendances. Published tables are freely available. digital.nhs.uk — HES

Health Survey for England (HSE) — NHS England / OHID Annual survey of approximately 8,000 adults and 2,000 children covering health behaviours (smoking, drinking, physical activity), measurements (BMI, blood pressure), and self-reported health conditions. digital.nhs.uk — HSE

Public Health Profiles (Fingertips) — NHS England / OHID Hundreds of health indicators at local authority level and above: life expectancy, mortality by cause, disease prevalence, health inequalities, and much more. fingertips.phe.org.uk

Cancer Registration Statistics — ONS / NHS England Cancer incidence by site, age, sex, and stage at diagnosis. England's cancer registration is among the most complete in the world. ons.gov.uk — Cancer Registration

Cancer Survival Statistics — ONS One-year and five-year cancer survival rates by cancer site, age, sex, and stage. Critical for assessing critical illness claim probabilities. ons.gov.uk — Cancer Survival

Quality and Outcomes Framework (QOF) Prevalence Data — NHS England Disease prevalence rates at GP practice level for diabetes, cardiovascular disease, COPD, mental health conditions, and many others. Covers most of England's GP-registered population. digital.nhs.uk — QOF

UKHSA Data Dashboard — UK Health Security Agency Public health surveillance data including COVID-19, respiratory viruses, vaccinations, and antimicrobial resistance. Replaced the original COVID-19 dashboard with broader health monitoring. ukhsa-dashboard.data.gov.uk

9. Income and Earnings

Annual Survey of Hours and Earnings (ASHE) — ONS The most detailed source of earnings data in the UK. Pay distributions by region, occupation, age, sex, industry sector, and full-time/part-time status. Based on a 1% sample of PAYE employees. ons.gov.uk — ASHE

Effects of Taxes and Benefits on Household Income — ONS Shows original, gross, disposable, and post-tax income by decile. Useful for understanding income redistribution and inequality. ons.gov.uk — Taxes and Benefits

10. House Prices

UK House Price Index (UK HPI) — HM Land Registry / ONS The official UK house price index. Average prices by property type and area, monthly price changes, and volume of sales. Coverage across all four UK nations. gov.uk — UK HPI

11. Inflation and Price Indices

Consumer Price Inflation Tables — ONS CPI and CPIH monthly time series. Detailed indices by COICOP category (food, housing, transport, etc.). Essential for inflation-linked benefit calculations in pension schemes. ons.gov.uk — Consumer Price Inflation

RPI Reference Tables — ONS Long-running RPI time series from 1947 to present. Still widely used for index-linked benefits in pension schemes despite the planned reform of RPI. ons.gov.uk — RPI

12. Interest Rates and Gilt Yields

Bank of England Yield Curves — Bank of England Nominal and real (index-linked) government bond yield curves estimated daily. Spot rates, forward rates, and implied inflation term structure. Essential for discount rate setting and asset-liability matching in pension valuations. bankofengland.co.uk — Yield Curves

Bank Rate History — Bank of England Complete history of Bank Rate (base rate) changes. Simple but essential for economic modelling. bankofengland.co.uk — Bank Rate

Bank of England Statistical Interactive Database — Bank of England Comprehensive monetary and financial statistics: money supply, lending data, exchange rates, effective interest rates. Flexible query builder. bankofengland.co.uk — Statistics

DMO Gilt Data — UK Debt Management Office Details of all outstanding gilts (conventional and index-linked), including coupon, maturity date, amount outstanding, prices, and auction results. dmo.gov.uk — Gilt Data

13. Student Loans

Student Loans Company Statistics — SLC / DfE Statistics on student loan borrowers, repayments, outstanding balances by repayment plan, and demographic breakdowns. gov.uk — Student Loans

Student Loan Forecasts — OBR Long-term student loan forecasts including expected write-offs, RAB charge, and repayment profiles by cohort. obr.uk — Supplementary Forecast Information

14. Census and Population Estimates

Census 2021, England and Wales — ONS Full results from the 2021 Census. Population counts by age, sex, ethnicity, religion, health, disability, housing, qualifications, occupation, and more. Available down to output area level. ons.gov.uk — Census

Scotland Census 2022 — National Records of Scotland Scotland's census was delayed to 2022. Provides similar detail to the England and Wales census. scotlandscensus.gov.uk

Northern Ireland Census 2021 — NISRA Census results for Northern Ireland at detailed geographic levels. nisra.gov.uk — Census

Mid-Year Population Estimates — ONS Estimated population by single year of age and sex for the UK and constituent countries. The standard population denominator used across government statistics. Updated annually. ons.gov.uk — Mid-Year Estimates

Small Area Population Estimates — ONS Population estimates at LSOA, MSOA, and ward level. Enables very granular geographical analysis. ons.gov.uk — Small Area Estimates

15. Migration and Births

International Migration Statistics — ONS Long-term international migration estimates by citizenship, country of birth, and reason for migration. Now primarily based on administrative data sources. ons.gov.uk — International Migration

Birth Statistics — ONS Live births by mother's age, area, and parity. Includes total fertility rate (TFR) and age-specific fertility rates (ASFR). Important for population projection and pension scheme demographic modelling. ons.gov.uk — Birth Statistics

International Datasets

16. International Mortality and Life Tables

Human Mortality Database (HMD) — UC Berkeley / Max Planck Institute The gold standard for international mortality research. Detailed mortality and population data for over 40 countries, with data going back centuries for some. Includes period and cohort life tables, death counts, exposures, and death rates by single year of age. Free registration required. mortality.org

Human Life-Table Database (HLD) — Max Planck Institute / INED Collection of life tables from official national statistical offices. Broader country coverage than HMD (160+ countries, 700+ life tables) but less standardised processing. Free registration required. lifetable.de

US Social Security Administration — Period Life Tables — SSA US period life tables showing probability of death, survivors, and life expectancy by single year of age (0 to 119) and sex. ssa.gov — Period Life Tables

US Social Security Administration — Cohort Life Tables — SSA Cohort life tables showing historical and projected mortality for US birth cohorts. Invaluable for long-term pension and annuity valuations. ssa.gov — Cohort Life Tables

Statistics Canada — Life Tables — Statistics Canada Complete life tables for Canada by sex at national and provincial levels. statcan.gc.ca — Life Tables

Australian Bureau of Statistics — Life Tables — ABS Australian life tables including state/territory breakdowns, Indigenous life tables, and trend data going back to 1881-1890. abs.gov.au — Life Tables

WHO Mortality Database — World Health Organization Cause-of-death data reported by 120+ WHO member states, classified by age, sex, and ICD cause. The most comprehensive international source for cause-specific mortality. who.int — Mortality Database

Eurostat — Mortality and Life Expectancy — Eurostat Life tables, life expectancy, and detailed mortality statistics for all EU/EEA member states. Accessible via Eurostat's data browser. ec.europa.eu — Eurostat Demography

SOA Mortality Tables — Society of Actuaries Comprehensive library of mortality tables used in US and Canadian practice, including CSO tables, RP-2014/MP-2021 pension tables, and mortality improvement scales. The mort.soa.org tool allows table lookup, comparison, and download. mort.soa.org

17. Insurance and Claims Data

FARS — Fatality Analysis Reporting System — NHTSA (US) Census of all fatal traffic crashes in the United States since 1975. Over 30 data elements per record, including crash circumstances, vehicles, and persons involved. nhtsa.gov — FARS

CRSS — Crash Report Sampling System — NHTSA (US) Nationally representative sample of all police-reported crashes (not just fatal), covering approximately 50,000 crashes per year. Replaced the former General Estimates System (GES). nhtsa.gov — CRSS

NAIC Insurance Data — National Association of Insurance Commissioners (US) Aggregate insurance market data including market share reports, profitability by line of business, and state-level premium data. Some reports are freely available. naic.org — Statistical Data

APRA Insurance Statistics — Australian Prudential Regulation Authority Quarterly and annual statistics on the Australian insurance industry: general insurance (premium, claims, combined ratios by class), life insurance, and private health insurance. Individual insurer-level data also available. apra.gov.au — Insurance Statistics

EM-DAT International Disaster Database — CRED, Universite catholique de Louvain Comprehensive database of natural and technological disasters worldwide since 1900. Over 26,000 disasters with data on deaths, injuries, affected population, and economic damages. Free registration required. emdat.be

NOAA Storm Events Database — NOAA (US) Records of significant weather events in the US from 1950 to present: tornadoes, hail, floods, hurricanes, and winter storms. Includes property damage estimates and fatalities. Over 48 event types tracked. ncei.noaa.gov — Storm Events

IBTrACS — International Best Track Archive — NOAA The most complete global collection of tropical cyclone track data, merging data from all regional meteorological agencies. Includes position, intensity, wind radii, and pressure for every known tropical cyclone. ncei.noaa.gov — IBTrACS

Insurance Information Institute Facts and Statistics — III (US) Free aggregate statistics on the US insurance industry covering auto, home, commercial lines, life, and health. Historical loss data and catastrophe loss summaries. iii.org — Facts and Statistics

CAS Loss Reserving Database — Casualty Actuarial Society Loss development triangles from NAIC Schedule P filings for multiple lines (workers comp, commercial auto, medical malpractice, other liability, private passenger auto, product liability). Widely used in academic research and reserving education. casact.org — Loss Reserving Data

18. International Health Data

CDC WONDER — US Centers for Disease Control and Prevention Wide-ranging US public health datasets including detailed mortality by cause, age, race, and geography. Also covers natality, cancer statistics, environmental data, and vaccine adverse events. wonder.cdc.gov

NHANES — National Health and Nutrition Examination Survey — CDC (US) Combines interviews and physical examinations covering demographics, dietary habits, health conditions, and biomarker data. Unique for providing objectively measured health data. Approximately 5,000 persons examined per year. cdc.gov — NHANES

NHIS — National Health Interview Survey — CDC (US) Principal source on the health of the US civilian population. Covers health status, health care access, insurance coverage, immunisation, and health behaviours. Approximately 35,000 households per year. cdc.gov — NHIS

MEPS — Medical Expenditure Panel Survey — AHRQ (US) Comprehensive data on health care use, expenditures, payment sources, and insurance coverage. Uniquely links medical conditions to specific costs. meps.ahrq.gov

WHO Global Health Observatory — World Health Organization Over 1,000 health indicators across all 194 member states: life expectancy, disease burden, risk factors, health systems, and expenditure. who.int — GHO

WHO Global Health Expenditure Database — WHO Internationally comparable health spending data for 190+ countries. Government, private, and out-of-pocket expenditure as shares of GDP and per capita. apps.who.int — GHED

OECD Health Statistics — OECD Comparative data on health systems across 38 OECD countries: health status, resources, utilisation, pharmaceutical markets, and expenditure. stats.oecd.org

Global Burden of Disease (GBD) — IHME, University of Washington Estimates of mortality and disability from hundreds of diseases, injuries, and risk factors worldwide. Covers 204 countries, 369 diseases/injuries, and 87 risk factors. vizhub.healthdata.org — GBD Results

CIHI — Canadian Institute for Health Information Data on Canadian health system performance including hospital outcomes, wait times, health spending, workforce, and patient safety. cihi.ca — Data and Reports

AIHW — Australian Institute of Health and Welfare Australia's national health and welfare statistics: health expenditure, disease burden, hospital statistics, mental health, aged care, and disability. aihw.gov.au — Reports and Data

Eurostat Health Database — Eurostat European health statistics covering causes of death, healthcare expenditure, resources, morbidity (hospital discharges by diagnosis), and health determinants. Standardised across EU member states. ec.europa.eu — Eurostat Health

19. International Demographics and Population

US Census Bureau — United States Census Bureau The definitive source for US population data. Decennial census results, American Community Survey (ACS) annual estimates, population projections, housing data, and economic census. Data available down to block group level. data.census.gov

US Population Projections — US Census Bureau National and state population projections by age, sex, race, and Hispanic origin. Base, high, and low variants. census.gov — Population Projections

Statistics Canada — Census and Population — Statistics Canada Comprehensive Canadian demographic data: census results (every 5 years), population estimates, projections, immigration, fertility, and household composition. statcan.gc.ca — Census

Australian Bureau of Statistics — Population — ABS Australian population statistics: census results (every 5 years), estimated resident population, projections, migration, births, deaths, and marriages. abs.gov.au — Population

Eurostat Population and Demography — Eurostat Comprehensive demographic data for all EU/EEA countries: population by age and sex, fertility, mortality, migration, household composition, and EUROPOP projections. ec.europa.eu — Eurostat Population

UN World Population Prospects — UN DESA The authoritative source for global demographic estimates and projections. Every country from 1950 to 2100. Fertility, mortality, migration assumptions with multiple scenarios. The 2024 Revision is the latest. population.un.org — WPP

UN World Urbanization Prospects — UN DESA Urban and rural population estimates and projections for all countries, 1950 to 2050. Includes data for individual cities and urban agglomerations. population.un.org — WUP

20. International Finance and Economics

FRED — Federal Reserve Economic Data — Federal Reserve Bank of St. Louis Over 800,000 time series covering interest rates (Treasury yields, SOFR), inflation (CPI, PCE), GDP, employment, housing, exchange rates, and international data. One of the most valuable free economic data resources in the world. fred.stlouisfed.org

US Bureau of Labor Statistics — BLS Primary source for US inflation and employment statistics. CPI at detailed item level, PPI, Employment Cost Index, occupational wages, workplace injuries/fatalities, and productivity data. bls.gov — Data

US Bureau of Economic Analysis — BEA GDP and national accounts data, personal income by state, and international trade statistics. bea.gov — Data

US Treasury Interest Rate Data — US Department of the Treasury Daily Treasury yield curve rates, bill rates, and real yield curve rates (TIPS). The foundation for US discount rate curves in pension and insurance valuations. home.treasury.gov — Interest Rate Statistics

World Bank Open Data — World Bank Over 1,400 development indicators covering GDP, poverty, health, education, infrastructure, and financial sector data for 217 economies. Includes insurance penetration/density data. data.worldbank.org

IMF Data — International Monetary Fund International Financial Statistics, World Economic Outlook data, Balance of Payments, Government Finance Statistics, and Financial Soundness Indicators for 190 member countries. data.imf.org

OECD Economic Data — OECD Economic data for 38 OECD countries: GDP, inflation, interest rates, unemployment, trade, productivity, and income inequality. The OECD Economic Outlook provides biannual forecasts. data.oecd.org

ECB Statistical Data Warehouse — European Central Bank Euro area and EU financial statistics: yield curves (government and corporate), money market rates (ESTR), bank lending rates, exchange rates, and financial stability indicators. ECB yield curves are essential for Solvency II discount rate calculations. sdw.ecb.europa.eu

Bank of Canada Financial Data — Bank of Canada Canadian interest rates, bond yields, exchange rates, and inflation data. Includes historical Government of Canada benchmark bond yields. bankofcanada.ca — Rates

RBA Statistical Tables — Reserve Bank of Australia Australian interest rates, bond yields, exchange rates, and economic indicators. Includes Commonwealth Government securities yields. rba.gov.au — Statistics

US Federal Student Aid Data — US Department of Education Federal student loan portfolio data: outstanding balances by loan type, repayment status, delinquency and default rates, and borrower demographics. studentaid.gov — Data Center

21. Pensions and Social Security

SSA Actuarial Publications — US Social Security Administration Comprehensive actuarial data on the US Social Security system. The annual Trustees Report provides 75-year financial projections, mortality tables, disability rates, and demographic projections. ssa.gov — Office of the Chief Actuary

SSA Disability Experience Data — SSA Disability insurance programme experience: incidence rates by age and sex, termination rates, and prevalence rates. ssa.gov — DI Statistics

PBGC Data Books — Pension Benefit Guaranty Corporation (US) Data on US defined benefit plans insured by PBGC: plan counts, participants, funding levels, premiums, and termination data. pbgc.gov — Data Books

Form 5500 Filings — US Department of Labor Annual return/report data for approximately 800,000 US employee benefit plans. Plan financial information, participant counts, funding data, and service provider fees. dol.gov — Form 5500

CPP Actuarial Reports — Office of the Chief Actuary, OSFI (Canada) Triennial actuarial reports on the Canada Pension Plan with 75-year financial projections, demographic and economic assumptions, and sensitivity analysis. osfi-bsif.gc.ca — CPP Reports

CPP Investments Reports — CPP Investments (Canada) Quarterly and annual reports on CPP fund investment performance and asset allocation. cppinvestments.com — Financial Results

APRA Superannuation Statistics — APRA (Australia) Comprehensive data on Australian superannuation funds: total assets, fund flows, asset allocation, investment performance, membership, and benefit payments. Individual fund-level data available. apra.gov.au — Superannuation Statistics

OECD Pensions Data — OECD Cross-country pension data: fund assets, contributions, benefits, replacement rates, retirement ages, and investment allocation. "Pensions at a Glance" provides comparative analysis. stats.oecd.org

EIOPA Data — European Insurance and Occupational Pensions Authority Solvency II risk-free interest rate term structures (published monthly), EU insurance market statistics, and occupational pension statistics. The risk-free rate curves are mandatory regulatory data for European insurers. eiopa.europa.eu — Tools and Data

22. Catastrophe and Climate Data

NOAA Climate Data — NOAA National Centers for Environmental Information Historical temperature, precipitation, extreme weather, sea level, and Arctic/Antarctic data. Climate Data Online provides station-level observations. Also produces the "Billion-Dollar Weather and Climate Disasters" dataset. ncei.noaa.gov

USGS Earthquake Data — United States Geological Survey Real-time and historical earthquake data worldwide. Comprehensive catalog with magnitude, location, depth, and shakemap data. Also produces seismic hazard maps. earthquake.usgs.gov

Munich Re NatCatSERVICE — Munich Re Aggregate natural catastrophe loss data including annual summaries, event counts, and trend analyses. The annual "Natural Catastrophe Review" is publicly available and widely cited. munichre.com — NatCatSERVICE

Swiss Re sigma — Swiss Re Institute Global catastrophe loss data and insurance market statistics via the sigma explorer. Complements Munich Re's NatCatSERVICE with independent estimates. sigma.swissre.com

23. Additional Specialist Sources

SOA Experience Studies — Society of Actuaries Industry experience studies covering mortality, morbidity, lapse/persistency, and annuity experience. Many provide detailed data tables. soa.org — Experience Studies

CIA Research — Canadian Institute of Actuaries Research papers and some experience data relevant to Canadian actuarial practice. Many research documents are publicly accessible. cia-ica.ca — Publications

Actuaries Institute Australia — Knowledge Bank Green papers, white papers, and public-interest research including the Australian Actuaries Climate Index. actuaries.asn.au — Knowledge Bank

ILO Social Protection Data — International Labour Organization World Social Protection Database covering pension, health, unemployment, disability, and maternity programmes for all ILO member states. ILOSTAT provides labour statistics. ilostat.ilo.org

NBER Business Cycle Dating — National Bureau of Economic Research The official chronology of US business cycles (peaks and troughs) since 1854. The definitive reference for recession dating. nber.org — Business Cycles

NCCI Workers Compensation Research — NCCI (US) Free research including the annual "State of the Line" report, industry financial results, and frequency/severity trends for workers compensation. ncci.com — Research

24. Quick Reference by Actuarial Specialty

25. Multi-Domain Data Portals

These portals are gateways to many of the individual datasets listed above, and much more besides. Worth bookmarking as starting points.

data.gov.uk — The UK Government's open data portal. Searchable catalogue of thousands of datasets across central and local government. data.gov.uk

ONS — The primary portal for UK population, economic, and social statistics. Topic-based navigation and expanding API access. ons.gov.uk

NOMIS — Labour market, population, and mortality statistics with flexible query tools. Particularly strong for area-based analysis. nomisweb.co.uk

UK Data Service — Access to survey microdata including the Family Resources Survey, Health Survey for England, and Labour Force Survey. Free registration required for most datasets. ukdataservice.ac.uk

Stat-Xplore — DWP benefits statistics including State Pension, Universal Credit, PIP, and Attendance Allowance. Free registration required. stat-xplore.dwp.gov.uk

26. Tips for Working with These Datasets

1. Combine exposure and event data. Use Mid-Year Population Estimates as denominators with death registration data as numerators to calculate mortality rates. Use road traffic statistics as exposure when analysing STATS19 accident data.

2. Geography matters. Many UK datasets use standard geographical classifications (LSOA, MSOA, local authority, region). The ONS Open Geography Portal at geoportal.statistics.gov.uk provides boundary files and lookup tables to link these.

3. Check reference periods. Different datasets use different time bases: calendar year, financial year, mid-year to mid-year. Always check the metadata before joining datasets.

4. Use APIs where available. FRED, Eurostat, the World Bank, IMF, Census Bureau, and many others offer well-documented APIs. The ONS is expanding its API offering via developer.ons.gov.uk. Programmatic access improves reproducibility.

5. Watch for registration requirements. HMD, HLD, EM-DAT, EPC data, Stat-Xplore, and UK Data Service all require free registration before download. Most other sources allow immediate access.

6. Data lag varies. Most government statistical datasets have a 1-3 year lag between the reference period and availability. Financial market data (FRED, ECB, Treasury yields) is typically available with minimal lag.

Found this useful?

I'll keep this list updated as new sources emerge. If you know of a free public dataset that should be on this list, get in touch at INQA Group — I'd love to hear what data sources you use in your work.

INQA Group is a community for actuaries who want to work smarter with AI and modern tools. If that sounds like you, take a look.

Last updated: February 2026 Compiled by Patrick Lee, Founder, INQA Group

my experience of intensive use of AI during 2025, and especially so far during 2026 (because the pace of improvement has accelerated!) is that AI is coming for white-collar jobs faster than predicted.  Blue-collar jobs too, but it will take longer to mass manufacture hundreds of millions of robots.  

Producing AI agents is much much quicker. 

So within perhaps 2 years at most, AI looks likely to be able to do almost all white collar work.  There will be exceptions, but those look likely to be temporary: e.g.

• people with specialist knowledge of a company's clients, data and systems (the AI will catch up eventually)
• people with roles mandated by regulation (regulators will be slow to act, but eventually will realise that AI will eventually be capable of doing a better job).

Many employers will be slow to react at first.  But the pressure not to recruit new humans (unless a strong business case can be made that the work can't be done by existing employees augmented with AI) will grow.  Employers that fail to adapt are likely to find that competitors outpace them.

So what should you do? 

Use AI to increase your productivity by a multiple.

That's what I have been working on over the past year. I have achieved it in several areas (most spectacularly in coding) and have been building a system that increases my productivity every day.

There are big societal issues coming too: as I put in a newsletter to International Qualified Actuaries Group members recently:

"We're heading toward an agentic world where AI agents can do most white-collar work within 2 years. Productivity becomes very high, products become cheap — but far fewer people work, including actuaries. Regulatory inertia will protect some temporarily, but not indefinitely.

The disruption goes deeper: What do pensions look like when lifespans extend but careers shorten? What's insurance's role with abundant goods but scarce employment? How do we finance universal basic income when traditional tax bases collapse? What's the point of professional exams when AI can do the work?"

The process was relatively straightforward: as well as changing the .csproj files, I also had to upgrade various Microsoft nuget packages. Note that for the function I couldn't upgrade Microsoft.ApplicationInsights.WorkerService to 3.0.0 but only to 2.22.0 because it seems that the .NET 10 functions are not yet compatible with Microsoft.ApplicationInsights.WorkerService 3.0.0.

At the moment (unless something else supplants them - not impossible given the pace of progress in AI) I think they will have a massive impact in 2025.

I also tried upgrading the associated function, but it seems (as too often with Microsoft releases) that .NET 9 is not yet really fully available.  It doesn't work with functions yet:

I tried increasing the version of the function to .NET 9 but I got a compile error:

"Invalid combination of TargetFramework and AzureFunctionsVersion is set."

I tried increasing the function version from 4 but 4 seems to be the highest currently available version.

So it seems that functions can't yet be upgraded to .NET 9.

Learn more

https://scorecard.inqa.com/neworg-1

For IFoA students:

https://scorecard.inqa.com/students-union-1

For IFoA qualified actuaries (Fellows and Associates), including those who left the IFoA

Are you an actuary looking for a new organization that avoids regulatory overreach, values transparency, network opportunities, and offers great value for money? Look no further! Join us in shaping a new future for current IFoA members by sharing your views and preferences. Take our quick quiz to join the waitlist and let your voice be heard: https://scorecard.inqa.com/neworg-1

#actuaries #competition #valueformoney #transparency

For IFoA students

See content credentials

Take our quiz to join the Actuaries Student Union Waitlist and share your thoughts on value for money, network, control, study, and representation! Click here to get started: https://scorecard.inqa.com/students-union-1 #Actuaries #StudentUnion #Quiz

(Students from other actuarial bodies, membership might also in due course be useful for you, but the Union will focus initially on meeting the needs of IFoA students).

#actuaries #competition #valueformoney #education #union

Is the Institute and Faculty of Actuaries (IFoA) going too far with its regulation of members? There has been a very active public debate here on LinkedIn over the past week see e.g.

• https://www.linkedin.com/posts/paulteggin_actuaries-code-dei-consultation-there-is-activity-7255515189968617472-FyMj?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/davidadamshaffer_dei-ifoa-actuaries-activity-7255536253746515968-OI9b?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/sueechiehtan_i-am-signing-paul-teggins-petition-to-object-activity-7254625837512024065-P9ox?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/hiteshshah100_ifoa-dei-diversity-activity-7255625469998333953-mzAv?utm_source=share&utm_medium=member_desktop

against the IFoA's proposals to embed controversial "DEI" into the Actuaries' Code and (largely in support of the proposals) IAA President (and IFoA Council member and Chief Actuary at Mercer) @Charles Cowling, Maria Kamenarova, PhD, FRM and @Leah Evans' comments on those posts (particularly in response to former IFoA President Tan Suee Chieh's powerful intervention).

Although it could have, the IFoA has not conducted a poll of its members. The only non anonymous poll (publicised to the largest actuarial group on LinkedIn and elsewhere) was 2 IFoA members for, and 42 IFoA members and 7 ex IFoA members against the "DEI" proposals.

In echoes of the significant unhappiness last year in the way the surprise governance changes were handled (when a similar number of senior actuaries wrote in objecting to the changes, or at least to the lack of transparency and proper consultation) 98 IFoA members and 7 former members have co-signed Paul Teggin's consultation response opposing the proposals.

It is increasingly clear that the IFoA is losing the hearts and minds of a significant number of experienced members. Unfortunately, it doesn't seem to have listened to the concerns raised last year. For anyone who wants to and hasn't yet, the deadline for responses to the "DEI" consultations is 29 October.

The IFoA acts as if members have no alternative. For many years they have been right.

Time for change

As you may know, I am working on setting up NewOrg, an actuarial organisation that is designed to go back to a time when an actuarial membership body concentrated on regulating that its professionals behaved with integrity and competence in their work, and didn't overreach into members' personal and political lives.

IFoA current and former members will be able to register their interest in that within a day or two at the latest.

#actuaries #competition #valueformoney #transparency

#actuaries #competition #valueformoney #education #union 

As some of you already know, I am working on a project to introduce competition into the monopoly that has been enjoyed for many years by the Institute and Faculty of Actuaries (IFoA).

Fellows and Associates

I know many are frustrated and even angry at the way the IFoA has behaved during the last two years:

• in 2023 with the poor way the governance changes were handled and communicated
• in both 2023 and 2024 with the way the IFoA and its Regulatory Board are persisting in pushing the controversial and political "DEI" ("Diversity Equity and Inclusion") doctrine into a key aspect of members' life: the Actuaries' Code.

Having spent 34 years as an IFoA member, including 8 years as an insider (on Council and Management Board and other Boards/Committees), I think it is fair to say that after qualification most actuaries used to regard the IFoA with a modicum of respect or at worst with indifference.

In recent years, that seems to have changed with many now either fearing the IFoA or even strongly disliking it.

The IFoA is felt by many experienced Fellows to be:

• overly controlling
• intrusive
• bureaucratic
• incompetent
• secretive, and
• poor value for money.

Some actuaries (some 10% of Fellows) have no choice but to be with the IFoA: those who by legislation need a Practising Certificate for their job.

Traditionally an unknown (because the IFoA do not disclose this) number of Fellows leave the IFoA before retirement, either because they feel that their job no longer needs IFoA membership (and they no longer wish to pay high fees to stay a member). For reasons that the IFoA said they wanted to research and publish (but have so far failed to do so), this happens disproportionately to female Fellows.

But increasingly I hear Fellows who are fully intending to continue working for many years saying "if only there was an alternative organisation, I'm really fed up with the IFoA".

These Fellows (who don't need a Practising Certificate) stay because they have felt that the only alternative is to not be a member of any actuarial body. They are reluctant to do for a variety of reasons including:

• fear that their employer or clients will be unhappy if they are no longer an FIA/FFA (or in some cases an Associate [the number of Associates is still very small compared to Fellows])
• fear that their professional indemnity insurance costs will increase
• possible loss of networking
• possible impact on their future career
• a general reluctance to avoid a leap into the unknown.

All this is understandable.

Students

While reddit forums are not going to be representative of all IFoA students, they are nevertheless a useful guide to trends in student views towards the IFoA.

It only takes a casual glance at posts on the main sub Reddit used by IFOA actuaries around exam booking, sitting and result times to see the massive level of frustration that students have about their experience with the IFoA.

Frequent pain points seem to be:

• technology failures (e.g. when booking exams, but particularly on exam result days, which are stressful enough without having to repeatedly try to login to a site that keeps crashing)
• things that make the exams unnecessarily difficult (questions mismatched with the tuition or past papers, unpredictability and wide spread between markers, time pressure that is unwarranted because rarely needed in actuarial work)
• having to compete against well organised online cheating groups (something that the IFoA inexplicably - and unforgivably in my opinion - exacerbated by holding staggered sittings of the same exam paper at different time intervals!)
• frequent changes to the exam formats and rules
• confusing rules about "plagiarism" which increases anxiety among the majority of diligent students about false positives (anecdotally several students have been falsely accused of plagiarism by the IFoA - although once again, the IFoA publishes little or no information about false positives and no statistics about alleged breaches of its "Assessment Regulations" - the rules students are subject to about exams)
• The very long and uncertain qualification time and the student dropout rate (again the IFoA publishes little or no information about this). My research shows that the time to reach Fellowship is increasing and has extremely high variability, and that about half of students who join the IFoA never make it to Fellowship.

Students who want to reach Fellowship (or Associateship) have little choice (unless the IFoA reintroduces lots of MRAs - Mutual Recognition Agreements - which in the past have allowed European actuaries to qualify often with only a handful of exams after university, as compared to the 13 that the IFoA currently requires). They have to stay with the IFoA until such time as they either qualify or decide that they have had enough of the exams.

The IFoA therefore has a very high level of monopoly power over students. How has it used it?

Well, if anything over the past few years the IFoA has arguably taken greater advantage of this power than before, not less.

For example:

• Despite boasting repeatedly in successive years about significant improvements to its website, the experience that students suffer on exam result days has generally remained dire
• The IFoA used to run student consultative forums which were at least a semblance of giving students a voice. But no student consultative forum minutes seem to have been published for several years, Even when minutes were published, students have complained that the minutes were heavily edited in favour of the IFoA.

How can we improve the journey for both students and Fellows/Associates?

What I am working on is a solution that will help both of these groups.

For Fellows and Associates who don't need a Practising Certificate

A new actuarial membership organisation [NewOrg] which they can join (either instead of, or in addition to the IFoA) which will:

• recognise the IFoA Fellow/ Associate qualifications and grant them Fellow / Associate membership of the new body
• record and monitor that members complete annual CPD requirements
• have a code of conduct (confined to professional life and proportionate and non intrusive, unlike the current IFoA)
• allow professional indemnity brokers/insurers/underwriters and clients/employers to verify that a member has the status they claim (Fellow / Associate)
• provide quality content including networking opportunities and online discussions
• for members who are still members of the IFoA, powerful facilities to organise
• for members who are still members of the IFoA, the mere existence of [NewOrg] will act as a strong catalyst for improved service and better value for money from the IFoA.

For IFoA students

IFoA students are stuck with the IFoA until such time as they either qualify or stop taking the exams.

As mentioned above the IFoA has enjoyed a near monopoly. In my opinion it has abused that power. Students have felt powerless and understandably fearful (note that practically all the posters on the main IFoA reddit sub group are anonymous). Occasional attempts have been made to organise petitions to send to the IFoA but these have usually been abandoned shortly after creation or been largely ignored by the IFoA.

History shows that there is a very powerful remedy to such an imbalance of power: a union.

What I have in mind is a union of IFoA students, for significantly less than the current IFoA student subscription fee. A union which will provide students with:

• the ability to organise and have strength in numbers
• a General Secretary (me) who will be their voice and pursue just demands and grievances with the IFoA.
• no longer will students need to personally stick their heads above the parapet with the IFoA. The Union and its General Secretary will do that for them.

And for those students who qualify: they will no longer be faced with a monopoly (the IFoA). Once they reach Fellowship they can choose between the IFoA and [NewOrg].

Interested? Register when the waitlist is available

If you are an IFoA Fellow, Associate or student with whom any of the above resonates, you will soon be able to register your interest in a waitlist. You can also message me here on LinkedIn if you want information earlier.

#actuaries #competition #valueformoney #education #union

Upgrading an app service (website) from .NET 7 (or 6) to .NET 8 is pretty straightforward and essentially just involves changing the target framework in the csproj file to <TargetFramework>net8.0</TargetFramework>.

Upgrading an Azure function from .NET 6 (or 7) in-process to .NET isolated is quite a bit more complex, with several "gotchas" which Microsoft have not documented very well.  You have to not only make several more changes to the csproj file, and change "FUNCTIONS_WORKER_RUNTIME" from "dotnet to "dotnet-isolated", but you also have to change the way dependency injection works (thankfully to be more consistent with app services - the differences between app services and functions were very annoying in the past).  The way logging works also needs to change, and that is very poorly documented at the moment.

Below is a comparison of the distribution of travel time to Fellowship of all 307 new Fellows in the sample with the distribution of the subset of 123 (out of the 307) who had taken an actuarial science degree or an actuarial science masters (or both).

The differences are very small.

The conclusion is that there is no evidence (at least from this sample of 307 Fellows) that actuarial science graduates or masters students qualify any quicker than students who do other degrees and don't do an actuarial science masters.

This is likely to be of concern both to the IFoA and the universities offering such courses. Once again I urge the IFoA to be transparent and publish a detailed study using its full dataset.

Methodology

I have carried out an analysis of what I hope is a representative sample from the 3,438 new qualifier records currently publicly available from the IFoA (Institute and Faculty of Actuaries) website. The latter are the new qualifiers from the 6 lists published from August 2020 to December 2022.

I have analysed 415 out of the 3,438 records.

The sample I have taken covers 108 out of the 584 new Associates and 307 out of the 2,618 new Fellows.

For each of these I have extracted (via a search on LinkedIn and/or the web generally) the likely starting date of work as an actuary (e.g. as an Actuarial Trainee, or Actuarial Assistant or similar). I have not taken the start date of an actuarial science degree or Masters, and I have ignored placements at actuarial firms during part of a degree course, and summer intern placements.

So for example, for someone doing a 3 year actuarial science degree followed by a one year Masters in Actuarial Science, I have taken the start date as the first date on their profile after the end of both of those.

I have then used the time between that start date and the start of the month of issue of the relevant IFoA new qualifiers list (e.g. 1 December 2022 for the December 2022 list). That is the travel time to Associate or Fellow, depending on which list the actuary appears in.

Important: because I am also interested in the time it takes to get from Associate to Fellow, I included in my sample of Associates and Fellows those who appeared as both of these in the 6 new qualifier lists. I will give the figures both including and excluding these "double" members in case that introduces a bias.

Headline results:

Associate Travel time: median 5.54 years, mean 6.71 years. Excluding double members: median 4.21, mean 6.18

Fellow Travel time: median 6.63 years, mean 7.42 years. Excluding double members: median 6.09, mean 7.23.

A fuller article coming later with results split by sex, country, whether or not the student did an actuarial science course, and university.

What if you want to store a copy of your settings (e.g. along with the artifacts) in a secure location for each version of the software you release?

There doesn't currently seem to be an easy way to export the values of the settings from the app service's Configuration view (Microsoft: why not? There should be!).

Instead, you can get it from the app service's Kudu website (Development tools, Advanced tools, Go).  On the Kudu site's home page, there is an App Settings link under REST API which gives you a json file with the settings and their values.

This will take you to a url like this (with https://[yourappservicename].scm.azurewebsites.net being the address of the Kudu site):

https://[yourappservicename].scm.azurewebsites.net/api/settings

Note

For some reason, the list does not seem to include settings which are in the ConnectionStrings part of the app service's Configuration.  Why not? Again this seems odd, Microsoft.

For information about EFCore Power Tools see https://www.youtube.com/watch?v=uph-AGyOd8c, That video is from 2020, but Erik has updated the tool (including on 31 Dec 2022!) and it has been downloaded over 250k times. He is https://twitter.com/ErikEJ on Twitter

<TargetFramework>net6.0</TargetFramework>

to 

<TargetFramework>net7.0</TargetFramework>

However, for some reason, although I asked in the publish profile from Visual Studio 2022 for it to publish using .NET 7, this didn't happen: I had to increase the setting by hand (Configuration, General settings) in the Azure portal for the app service to use .NET 7.

This was for the app service.  There is also an Azure function (which updates an Azure Table every day so that the data that is used to create the category and tag clouds is kept up to date).  It is not so clear that the function should be upgraded from .NET 6 to .NET 7 because .NET 7 functions are meant to run using the .NET Isolated Worker model, which could be a more significant change than that just upgrading the target framework for an app service. 

.

1. If the older code uses (or can be upgraded to use) Entity Framework 6.3.3 or later, then as an initial step you can keep the entity framework code largely unchanged, and just change the other code to .NET 6 or 7. This is because EF 6.3.3 works with .NET Core (see this article).
2. Once you have got rid of the .net framework code and replaced it with .NET 6 or 7, at a later date you can migrate the code from using Entity Framework to Entity Framework Core.

Error : AADSTS1002016: You are using TLS version 1.0, 1.1 and/or 3DES cipher which are deprecated to improve the security posture of Azure AD. Your TenantID is: [Guid]. Please refer to https://go.microsoft.com/fwlink/?linkid=2161187 and conduct needed actions to remediate the issue. For further questions, please contact your administrator.

There was no obvious reference to TLS in my apps and I didn't find the articles by Microsoft on how to solve the error particularly helpful (e.g. see https://learn.microsoft.com/en-us/troubleshoot/azure/active-directory/enable-support-tls-environment?tabs=azure-monitor). I eventually solved it by realising that the call to ADB2C was being made by an old WebAPI service that still used .net framework 4.5.2.  I was going to migrate this to .NET 7, but then tried just migrating it to .NET framework 4.8 and that worked.  The reason is probably that the instance that Microsoft was hosting the 4.5.2 app service on was probably running an older version of Windows that used TLS 1.0 or 1.1.

So if you get that TLS 1.0/1.1 error message, consider this as a possible reason.

If you revisit such apps now, you are informed that you need to migrate them to CLU (Conversational Language Understanding) but at present the migration seems to be buggy with repeated errors.  (Please fix this Microsoft).

However the original LUIS apps still seem to work (for now at least).

These normally include:

• a front end (either web, mobile or desktop)
• an authentication layer (e.g. Azure ADB2C - business to consumer)
• WebApis (either - if called directly by the user - requiring the user to be logged in via ADB2C, or - if called by another component - they could instead be protected via an APIKey, with other useful information transmitted in the headers)
• Azure functions (for serverless, scaleable computing)
• Azure SQL databases (for structured storage)
• Azure Storage accounts (for unstructured storage, e.g. blob files for images, json files, pdf reports, logs)
• Azure Cosmos Db (for more expensive unstructured storage requiring more search facilities)
• KeyVaults (to store secrets such as database connection strings).

Such diagrams are essential tools to help other developers understand how the system works.

It has libraries for Azure and Office that has suitable shapes to include in your diagram (see the screenshot at the top of this post for some examples).

You can do this for WebAPI by adding a GetNameAndVersionInfo method that can be called (and only needs to be protected by an APIKey rather than requiring a user to be logged in).

But what to do for Azure Functions? You could add a subfunction triggered by a web hook, but in order to have a record of different versions, I prefer to use a LogVersionInfo function triggered by a timer, normally daily.

You can also trigger it on demand (e.g. just after a deployment to check that the expected version has been deployed) by using the Code + Test section for the sub function in the Azure portal., and clicking the Test/Run link. (For some reason, you sometimes need to run it twice - another area where the Azure portal is a bit buggy that Microsoft should fix).

If as sometimes happens, you find gaps in the monitor's records, to check whether the function actually ran on a particular day, check the logs for the function: they are usually far more complete.

Of course if sampling is being applied to the Application Insights resource that the Azure Function is linked to, then some of your non error traces could be missing even in the logs.  If so, then you may need to either amend the settings for AppInsights (to avoid sampling being applied to particular types of traces), or you might need to pay for a more expensive AppInsights resource.

(NB: the link is to a zip file containing an Excel xlsm file.  Because that contains VBA code, it may be blocked by Microsoft Windows.  If so, to unblock it, see https://support.microsoft.com/en-gb/topic/a-potentially-dangerous-macro-has-been-blocked-0952faa0-37e7-4316-b61d-5b5ed6024216).

I have produced this in an attempt to help actuaries and others try and find out some key information which the IFoA has not to date been very transparent about:

-the student drop out rate (i.e. what percentage of students never complete the exams)

-what percentages of students qualify in what time (if they qualify)

- what the Fellow drop out rate is (i.e. what percentage of qualified actuaries leave the IFoA before retirement).

The model has a ReadMe page with instructions:

Screenshot of the inputs (and the button to run the VBA program):

Screenshot of the ModelOutput worksheet (which contains the initial population of students and fellows projected forward 100 years (which should be long enough to reach a steady state unless a long initial period of growth has been selected):

Screenshot of the PivotChart worksheet which is linked to the ModelOutput (its data source needs to be refreshed after a run).  You can change the fields and filters you want to look at here.

I invite fellow IFoA members to play around with the model.  It seems very difficult to put in decrement rates that fit with the IFoA's past assertion that the average qualification time is 7 years (or 5 years has also been mentioned) and also with the published membership student and Fellow population profiles that appear from time to time on the IFoA's website.

The best thing would be for the IFoA to publish detailed statistics on student qualification times, student and fellow drop out rates of course.

Well although the link in the Azure portal to App Insights is not disabled for a .NET 6 web app, you can still find that despite everything appearing to be properly configured within the portal, no data (requests, traces, exceptions etc) appear in the App Insights logs.

As with my previous post on .NET 5, the solution seems to be to right click the project within Solution Explorer in Visual Studio and configure App Insights from there.

That adds code within the start up (in Program.cs for a .NET 6 web app) which seems to wake up whatever was asleep in Application Insights.

If you want a registered app of yours to be able to change users' passwords then although you can add, edit and delete user profiles via granting the app API permissions within Microsoft Graph, the app can't change users' passwords (at least not with application permissions [running as the application]).

Instead, you can add, edit and delete users, and also change their passwords (via the Update [patch] method on the user, either via the Graph API SDK or via calling the REST API) without any Graph API permissions, but instead giving your app an Identity (managed identity) and giving that identity User Administrator role.  This can be used within Azure Active Directory, or an ADB2C directory.

(Adding a role to a new app is easy. Finding out which roles an existing app already has is not easy - see this post).

At present, the user interface is terrible to find role assignments in the Azure portal: there does not seem to be any way to find the roles assigned to a particular registered app.

Instead, you have to select a role and then you can find which apps have been assigned to that role. But there are dozens of roles, so you would have to try them all to see which roles a particular app has!

Microsoft should fix this by adding a Roles link when you select a particular registered app, showing the roles that have been assigned to that app.

Leave a trail (for yourself and the rest of the team)

So if you add a role to an app, to make this self-documenting, add a description to the app in the Internal notes section:

This registered app is used. It does not need any API permissions (no API permissions allow user passwords to be changed). Instead it just needs User Administrator role.

The existing tools within Windows 10 didn't help as they don't make it easy to find out where the large files and folders are, and the tools on the disk don't seem to cover clearing user/AppData/Local.

The free version of TreeSize (*) was very useful in identifying the problem: OneNote backup and cache folders.  (I use OneNote extensively, as a sort of personal StackOverflow for recording work done, problems encountered and how solved).

https://superuser.com/questions/1468754/onenote-reducing-storage-footprint-on-pc was very helpful in this regard, I was able to change the location of OneNote's cache and backup folders to a separate fixed drive within the pc.

Update * : I have just bought the paid for version as a thank you to the JamSoftware team for helping me identify the problem.

(For some reason, Microsoft don't link to the data movement library from their page about AzCopy - they should do).

See here for AzCopy and here for the Data Movement library.

The official documentation (see https://docs.microsoft.com/en-us/azure/azure-functions/functions-host-json#functiontimeout) says that you can increase (or reduce) this by adding a functionTimeout value to host.json.

But this doesn't currently work for a .NET 6 Azure function (v4)  (the latest version)!

Instead, (my thanks to https://github.com/paulbatum who disclosed this trick on https://github.com/Azure/azure-functions-host/issues/6111) adding this setting AzureFunctionsJobHost__functionTimeout in app.settings.json (or in Configuration in the portal) works, e.g. for 23 hours:

"AzureFunctionsJobHost__functionTimeout": "23:00:00"

In the past I have had to use a standard Azure function to trigger a Durable Entity, but recently discovered that this isn't necessary.

Instead (see https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-http-api) Azure provides webhooks that you can call.

First, create the entity by calling an operation on it via a POST call

You first need to create the entity by calling an operation on it, via a url similar to the following: 

POST /runtime/webhooks/durabletask/entities/{entityName}/{entityKey}
    ?taskHub={taskHub}
    &connection={connectionName}
    &code={systemKey}
    &op={operationName}

That will both create the instance of the Entity (of type entityName and with uniqueId entityKey) and call the operation called operationName on it.

It should return a 202 Accepted code.

Query the state of the entity (e.g. progress) via a GET call

Once the entity has been created you can then get the exposed state of the entity via a url similar to the following

GET /runtime/webhooks/durabletask/instances/{instanceId}
    ?taskHub={taskHub}
    &connection={connectionName}
    &code={systemKey}
    &showHistory=[true|false]
    &showHistoryOutput=[true|false]
    &showInput=[true|false]
    &returnInternalServerErrorOnFailure=[true|false]

This should return a 200 OK code, with a body showing the publicly exposed properties of the entity.

Conclusion

Using these built in webhooks means that you can avoid having to create a separate (non durable) Azure function and an associated trigger for that function.

Very neat!

Last year it was Programming in C# (certification: here), and this year it was Microsoft Certified: Azure Developer Associate (certification: here).

These exams were online, but unlike the Institute and Faculty of Actuaries' online exams (which are un-proctored with allegations of significant cheating going on by a minority of students), these Microsoft exams are proctored using Pearson Vue.

I really don't understand why the IFoA has not done something similar: it is unfair to the majority of conscientious students (and to everyone who has already taken and passed the exam without cheating) to allow a significant minority to cheat. 

The Institute and Faculty of Actuaries has recently upgraded its website, with a new look.

Unfortunately, the search facility is far worse than it was before.  For example, if you search for "member pledge"  you currently get this result:

We couldn't find a match for "member pledge". Please try another search.

So for practical purposes the signposted search facility within the IFoA's new website is of little use: you will miss relevant documents, or worse (as we saw with the member pledge) think that no relevant documents exist.

Two ways to still have a good search experience on the site

The first is that you can (for the moment, unless they remove it) still go to their old site search facility at https://www.actuaries.org.uk/search/site and enter in your search term there.  

E.g. entering member pledge is equivalent to going to https://www.actuaries.org.uk/search/site/member%2520pledge and that brings up lots of results, including the Member Pledge page, and on the right hand side some options to filter the results down further.

The second is a standard way to use Google to search within a site.  In Google's search bar, just prefix your search term by site:urlofthesite, e.g. here site:actuaries.org.uk

So a search for member pledge is 

site:actuaries.org.uk member pledge

which is equivalent to https://www.google.com/search?q=site%3Aactuaries.org.uk+member+pledge&oq=site%3Aactuaries.org.uk+member+pledge

which brings up lots of pages: 

It uses the kusto querying language, which is subtly different to SQL so it takes a bit of getting used to in order to get the queries you want.  (NB you might find this 

Kusto to SQL cheatsheet useful).

Kusto query to get top items in descending order

Here is a very easy way to get a table (or chart, but the urls are hard to see in the chart because they are too long for a chart axis marker) of the top requests made to a website using kusto:

requests 
| summarize count() by url
| order by count_

You can filter the requests, e.g. to ignore the home page, or to exclude robots.txt by putting a | where query before the summarize operator

requests 
| where url != 'https://pjlee.net/' 
   and url != 'https://www.pjlee.net/'
    and url != 'http://pjlee.net/'
    and url != 'http://www.pjlee.net/'
    and not(url contains '.ico')
    and not(url contains '.js')
    and not(url contains '.css')
    and not(url contains '.txt')  
    and not(url contains '.php')  
| summarize count() by url
| order by count_

You can run this in the Application Insights logs page for your site (selecting a suitable time period), and then pin it to an Azure dashboard.

You can also get charts of counts of requests summarized for each day and include those on the same dashboard.

Improvements

I really like the AI enabled Intellicode feature: it suggests code as you type and if you like the suggestion you can accept it by just pressing the Tab key.  This works quite well and saves quite a lot of typing compared to Visual Studio 2019.

I haven't really noticed/used any other improvements yet other than Hot Reload: the ability to change code while debugging and continue without having to completely restart the debugging session.

Oddities/Bugs

Visual Studio 2022 quite often gives false positive error messages (i.e. error messages when there really hasn't been an error), either when publishing to Azure, or when pushing changes to Git. 

If you want to do code migrations to an Azure SQL database in a project that is still using .NET Framework and Entity Framework (rather than  Entity Framework Core) using the Nuget Package Manager Console, then I found I had to revert to using Visual Studio 2019.  Visual Studio 2022 kept on giving errors.

But overall, the AI Intellicode means that I like Visual Studio 2022 .

One thing I hadn't done is remove the usage of Startup.cs file.

.NET 5 websites by default have both a Program.cs and a Startup.cs file.

.NET 6 websites only have a Program.cs file.

I had updated the site to .NET 6 merely by changing the target framework in the project's properties), but I had left the existing Program.cs and Startup.cs files unchanged.

Removing the Startup.cs file

Today I wanted to see if I could remove the Startup.cs file. 

It turned out this was relatively easy for my project:

I first replaced the contents of Program.cs by the contents of the Program.cs of a standard new .NET 6 MVC website that I added temporarily to my Visual Studio 2022 solution. 

I then got a compile error on the very first line of Program.cs:

The name 'WebApplication' does not exist in the current context.

Comparing the contents of my project's  .csproj file with that  of the standard new .NET 6 project showed that the latter uses ImplicitUsings.

Adding  

<ImplicitUsings>enable</ImplicitUsings>

to the PropertyGroup in my project file solved the compile problem.

After that, I just had to add some services (previously in Startup.cs) in Program.cs (NB the syntax in Program.cs is slightly different: builder.Services.Add ..) and I was then able to delete the Startup.cs file.

It wasn't very difficult to upgrade the web app project from .NET 5 to .NET 6 (just changing the target framework). but I also updated various Nuget packages and changed from  Microsoft.Azure.Cosmos.Table (shown as deprecated) to Azure.Data.Tables, which took quite a bit of time.  (Why do Microsoft make changes so often: the last time was < 2 years ago from Microsoft.WindowsAzure.Storage to Microsoft.Azure.Cosmos.Table, and with the code needing to change in a far from obvious way?)

I also had some problems with the function. Although it seemed to upgrade from .NET Core 3.1 to .NET 6 fine (after upgrading the functions SDK) and it built ok, I got an error at run time. I had to remove some extra tmp csproj file that was there, which seemed to maintain a link to .net core 2.2.  After that it seemed to build and run ok.

This is because if there are n tasks which need to be orchestrated (in the chaining example at least), the orchestrator will perform 1 + 2 + 3 + 4 + ....  + n steps because of the way it "replays" each task by checking on each pass whether it has completed, and if not running it.

So this means n(n+1)/2 = n^2 + n/2 steps, which beyond say 500 tasks introduces quite a big overhead in running time.

So if you have a very large number of tasks, then it may be better to use older techniques for long running tasks, e.g. running as a WebJob, or running on a virtual machine, or sending each task to a function triggered by a queue, or using Azure Batch.

However, I have found that you can use Application Insights with a .NET 5.0 website, you just have to configure it from within Visual Studio 2019.

Doing that will add the Microsoft.ApplicationInsights.AspNetCore v 2.15.0 nuget package to the project, and it will add the telemetry service to the Startup class:

      public void ConfigureServices(IServiceCollection services)

        {

            services.AddControllersWithViews();

            services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);

        }

After that when you publish the website to Azure, although the Applications Insight link in the left hand menu is still disabled, there will be another link near the top of the Overview view which you can click to configure/view Application Insights.

I have found this reasonably straightforward in a number of projects, with some exceptions:

• There is still no .NET 5.0 version of Azure functions so they have to remain .NET Core 3.1 for the time being
• in some cases, I have had to stick with the 3.x versions of Entity Framework Core, rather than the 5.x versions.

More when I have time.

At times, I'm told that working conditions are unsanitary or even unsafe.

I have set up a new website, https://courtingcatastrophe.com where professionals involved can log problems at such venues, in the hope that this will help bring about safer/better working conditions.

The site is designed to allow users to report problems using a smart phone or tablet, provided they are connected to the internet.

This version allows longer delays between infection, symptoms appearing, cases being reporting, and deaths. 

For example, to connect to blobs (binary large objects, typically images or files), queues (for distributed processing or to enable web apps to scale by decoupling front end from time consuming backend processing), or tables (for unstructured data storage).  

For blobs, queues and files

Use the new Nuget packages: Azure.Storage.Blobs (e.g. I have just used version 12.4.1), Azure.Storage.Queues and Azure.Storage.Files.Shares respectively.

For Tables

Azure Storage Tables provide a cheaper (but less powerful) way of storing relatively simple unstructured data as opposed to the more powerful (and expensive) facility in Azure CosmosDb.  

For some reason, there isn't an equivalent package for Azure Storage Tables. Instead, rather than using the previous WindowsAzure.Storage (version 9.3.3) Nuget package, you can use the Microsoft.Azure.CosmosTable Nuget package to connect to Azure Storage Tables.

Example of using Microsoft.Azure.Cosmos.Table to connect to an Azure Storage Table

I used version 2.0.0 preview (version 1.0.6 also works) of the package, but an earlier version may also work, I haven't tried this, (There doesn't seem to be any online documentation specifically for this, only for connecting to a CosmosDb table, but similar code seems to work for an Azure Storage Table).

The code below was for an ASP.NET Core 3.1 api web app:

For an Azure storage Table called MyTable, with a corresponding class called MyTableEntity:

using Microsoft.Azure.Cosmos.Table; // added for TableEntity

    public class MyTableEntity: TableEntity

    {

        public MyTableEntity(string type, string id)

        {

            this.PartitionKey = type;

            this.RowKey = id;

        }

        public MyTableEntity() { }

// put your table's other properties here

    }

then in a controller:

    using Microsoft.Azure.Cosmos.Table;

      private async Task<List<MyTableEntity>> GetMyTableEntities()

        {

            var myTable = GetMyTable();

            // Construct the query operation for all entities where PartitionKey="putpartitionkeyvaluehere";

            TableQuery<MyTableEntity> query =

                new TableQuery<MyTableEntity>().Where(

                    TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "putpartitionkeyvaluehere"));

            var records = new List<MyTableEntity>();

            TableContinuationToken token = null;

            do

            {

                TableQuerySegment<MyTableEntity> resultSegment = await myTable .ExecuteQuerySegmentedAsync(query, token);

                token = resultSegment.ContinuationToken;

                records .AddRange((resultSegment.Results));

            } while (token != null);

            return records ;

        }

        private CloudTable GetMyTable()

        {   // use dependency injection to set _config (the configuration) in the controller's constructor or use KeyVault or ManagedIdentity to avoid having to store secrets in settings

            var connectionString = _config.GetValue<string>("AzureStorage:ConnectionString"); 

            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);

            CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

            CloudTable myTable = tableClient.GetTableReference("MyTable");

            return myTable;

        }

See the section entitled "Update 1 April 2020" on this page of David's website. On that page, David produces full details of his model, including documentation and calculations in Excel.

The model projects the following peak dates (taken as those when the number of deaths is at its maximum), and eventual total number of deaths, under 9 different scenarios: 

(Please note that these are just from the current outbreak, so ignoring possible further outbreaks later this year or next year. They also ignore any possible overrun on the UK's National Health Service).

Scenario A is for illustration on the assumption that the mitigation policy in place before 21 March had remained in place. We know that the Imperial College Covid-19 Response Team's equivalent projection is about 510,000 deaths, so the two models are at odds here. We have to hope that the Imperial College model is more correct, but it seems to me that the challenge that David's model poses at the moment is: it seems coherent, so if it is wrong, in which areas? I have not seen the Imperial College model. Of course, with an epidemic where the number of cases seems to be doubling roughly every 4 days, it may be very easy to over or under-estimate the likely total number of cases by 2 or more, particularly when the true number of cases is unknown.

Scenarios H and I are the most optimistic ones as to the effectiveness of the UK's social distancing policy.

In scenario I, it is assumed (for the sake of showing one extreme) that the virus is immediately unable to spread further and dies out within 2 weeks, as all persons currently infected either recover or die. New cases are still detected for the first 5 days in line with the default parameters in this version (3) of David's model under which it is assumed that it takes 5 days for symptoms to appear.  

In scenario H, the virus spreads very slowly, and dies out by about mid June.

However, both of these scenarios already look extremely optimistic because the actual numbers of confirmed cases and deaths have already significantly exceeded their projections by the end of March.

Similarly the numbers of cases and deaths projected under scenarios E to G look implausible in comparison to the actual published numbers by 1 April.

But maybe we are in scenarios somewhere between A and E?  

More importantly, what is it about the modelling that the UK government is relying on that is leading them to predict a peak of new cases (and probably daily deaths) many weeks earlier than the projections in David's model? 

As David says, his model or calculations may well be wrong (and indeed he hopes it is wrong with regard to the high number of deaths projected, and would be delighted if someone can point out a mistake, or implausible assumption!).

I hope we (the collective human mind, pooling all our skills and experience together) ought to be able to examine the model relied on by the UK government in the same way: with exponential growth, two weeks can make the number of cases grow more than ten fold.

I understand that the source code may be being made available very soon by Professor Neil Ferguson.

For those unable or unwilling to log in to Linked In, here are recent bulletins published by the group in our (the whole world's) collective battle to make sense of and overcome the Covid-19 pandemic:

Covid-19 Actuaries Response Group Bulletin 4: UK is not like China! (30 Mar 2020).

This paper casts doubt on the conclusions of a paper from the Department of Electrical and Electronic Engineering of Imperial College suggesting that the total number of deaths from COVID-19 in the UK could be as low as 7100.

Covid-19 Actuaries Response Group Bulletin 5: Suppression v Mitigation (31 Mar 2020)
The summary from this paper is: 

Suppression strategies seek to reduce the spread of this epidemic by aggressive identification of infected individuals, with strict isolation of them. Innovative contact tracing and testing, has shown success in slowing the epidemic in countries like Singapore, China, Taiwan and South Korea. However, this leaves the population without immunity and susceptible to recurring outbreaks.
Mitigation strategies seek to manage spread of infection through a population by significantly reducing social interaction between and among population subgroups. Specific isolation measures are implemented for vulnerable subgroups expected to be most adversely affected by infection. Success depends on:
- getting timing + level of reduced social interaction right so health services are not overwhelmed,
- high levels of compliance with social distancing,
- successful identification and isolation of vulnerable subgroups, and
- acquisition of long term immunity by those who have been infected

Covid-19 Actuaries Response Group Bulletin 6: Modelling and the Covid-19 Pandemic (1 Apr 2020)
This paper gives further background to that from Bulletin number 5 on the choice between mitigation and suppression strategies, and discusses some aspects of some models produced so far.

My comments:

1. The numbers of deaths quoted in the Bulletin are the ones from the Imperial College Covid-19 Response Team's 16 March 2020 paper. As I have pointed out (see https://pjlee.net/blog/covid-19-some-observations-following-the-publication-of-the-imperial-college-response-teams-paper-of) the number at the top end of the range (510,00 deaths in the UK) understates, perhaps to avoid alarming the population, the true likely number of deaths, because (as the 16 March 2020 paper states, but doesn't change the numbers to allow for) that is on the unrealistic assumption that everyone who needs hospitalisation can get treated.  In reality the likely number of deaths could be up to 5 times as many (2.5 million), since the Imperial College paper assumes a death rate of 0.9% even if 4.4% of cases are able to receive hospital treatment.
2. It would have been useful if the paper had included a reference to David Wilkie's model, which meets the final exhortation in the paper ("encourages members to engage with international modelling community by sharing models, expertise and experience"). David's model is available for anyone to use.

Click here to use the web model. (NB It looks as if you might need to accept cookies on the site for the model to work.) It is not as flexible as the Excel files in that you can't change formulae (only the parameter values), but it is quick to use and to see charts of infections and deaths each day, and their cumulative totals over time.

Setting the Parameters

The screenshot below shows the default parameters:

After clicking the Run button you can look at the Chart and Result tabs:

Chart tab: showing and hiding different series

By clicking on the name of a series in the legend, you can toggle between showing that series or not showing it. In this way you can look in detail at each of the series, even though they have different orders of magnitude:

Under the default parameters (which do not reflect social distancing) the eventual number of cases gets very high by mid June:

After clicking the Cumulative Cases and Cumulative Total Deaths series in the legend, the next screenshot shows the peaks in cases (and deaths) are projected to occur in late May in the absence of social distancing:

David writes:

I am an actuary, quite familiar with epidemiological models, which can be used in income protection insurance (IPI), which many doctors/GPs will be familiar with.  Using the IPI methodology I have developed a simple, but realistic, model for Covid-19 which can be run by anyone familiar with using Excel and goes onto one worksheet.  No lengthy complicated programmes. I attach the current release of files, which you can use or forward to anyone who wishes:

I am always adding: 

“This model may be of use to those who are comfortable using Excel and wish to play with projections for Covid-19.  So they may not appeal to all.  I am not making forecasts of what will happen. This is a tool that allows one to put in parameter values and see what projections come out.  Don’t be depressed by the results.  Things may well turn out much better than some of these projections."

 David Wilkie (contact email: david.wilkie@inqa.com)

The files are (you can also check each file for integrity - namely that it hasn't been modified via its MD5 checksum, see http://onlinemd5.com/ for example):

Cv19DailyModel_V2.2.docx

(MD5 checksum: 75BADBA2CD794EB796ED99D191135D37)

CV19DailyModelNotesOnUK_02.docx

(MD5 checksum: EF8A85D1626981BDFC59516D29276F8A)

Cv19DailyModel-V2.2.xlsx

(MD5 checksum: 174E8B545B4C3D6C2B78A51A0CD0F766)

Cv19DailyModel_UK_02.xlsx

(MD5 checksum: A69EDCD6D643EFEEB301DEAD60FBB37B)

COVID-19-ECDC-2020-03-26.xlsx

(MD5 checksum: 4AE04629467813487F02DF6D42B4B9EE)

For those unable to or not wanting to log in to LinkedIn, Bulletin 3 from the Covid-19 Actuaries' Response Group can be downloaded from here.

Has over half of the UK population already been exposed to Covid-19?

My (InQA Limited and before that Watson Wyatt) colleague Professor David Wilkie has produced an Excel model to help actuaries and any one else who is interested model the possible spread of #Covid-19, and has asked me to put the files here so that they are readily available without anyone having to log in to Linked In or another platform. The model is in Excel so that the parameters can be readily altered.

Model description

David writes:

I have produced a daily version in one Excel worksheet.  I attach my description of the model (Cv19DailyModel_V2.1.docx), the accompanying basic Excel (Cv19DailyModel_V2.xlsx), my own trials on the UK data (Cv19DailyModel-ADW01.xlsx) and a file of data extracted from the European Centre for Disease Control (ECDC) website and processed into a tidy form (COVID19-Countries-2020-03-24.xlsx).  Note that ECDC is a day different from the UK.  The latest data in their file is dated 24 March, but published in the UK on 23 March, so is really perhaps cases and deaths noted on 22 March.

Make use of this if you wish, and pass it on to anyone that might be interested. My actuarial approach seems different from the epidemiologists, but I don’t have the mathematics of their model available.  It is easy to recalibrate for any other country.

David Wilkie

contact: david.wilkie@inqa.com

Please note the final paragraph of David's paper describing the model:

I must emphasise that I am not making specific forecasts.  I am providing a tool that allows others to try out the results of hypothetical changes in what seem to me to be a plausible set of initial assumptions.  If I made it much more realistic by stratifying the population by age, the model would need a full programme, which would be much harder for people to play with.  Those who are happy changing the Excel may modify it as they wish.  Others may just try altering certain values.  My only request is that you recognise the authorship, and do not try copywriting it yourself to prevent others using it.

Copyright A D Wilkie

24 March 2020

Model files

The files (see David's note above for what each file does; you can also check each file for integrity - namely that it hasn't been modified via its MD5 checksum, see http://onlinemd5.com/ for example) are: 

Cv19DailyModel_V2.1.docx 

(MD5checksum: 99836539D23ADF0B42DBB957815EB145)

Cv19DailyModel-V2.xlsx (MD5 checksum:  F912ABCCF9B8BB16D595220F0A965CB9)

Cv19DailyModel-ADW01.xlsx (MD5 checksum: EC6F4275ADC157B18DA8FD89FA18A46F) and

COVID19-Countries-2020-03-24.xlsx (MD5 checksum: 905F671BEB769D8F2A0D24C63D16256A)

Some people have asked to be able to read the Covid-19 Actuaries Response Group's paper of 24 March 2020, written in response to the Imperial College Covid-19 Response Team's paper of 16 March (see previous post), which is otherwise only accessible via logging in to LinkedIn.

A copy of the Actuaries Response Group's paper of 24 March may be downloaded from Download ActuariesCovid19SupportGroupPaper24Ma.... (I stress that I am not a member of this group, I merely follow it and have commented on it).

The UK government has stated that they are taking decisions based on scientific and medical advice. In particular, recent decisions have taken into account the paper published by the Imperial College Covid-19 Response Team on 16 March 2020 (see https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf).

The publication of that paper is a very welcome and considerable improvement in transparency which in my opinion is a very important part in:

1. The UK public (and people and governments of other countries since this is a global pandemic) maintaining confidence in the government's response to the Covid-19 crisis. 
2.  Equally importantly, enabling a truly scientific approach to work: without being exposed to scrutiny and criticism by as wide an audience as possible, flaws in the decision process are far less likely to be found.

(We are not at war with an intelligent enemy intent on reacting to our strategy, hence there should be no need for secrecy.  If any secrecy were to be involved, then the approach taken would no longer be scientific.)

I have now had time to read through the paper a couple of times and make some observations here and pose a few questions that jumped out at me.

(Note that the paper also contains modelling projections and similar conclusions for the USA, but here I focus on the implications for the UK).

Caveats

I am not trained in medicine or epidemiology.  My qualifications are in mathematics (a degree), statistics (a diploma in mathematical statistics), actuarial science(I am a Fellow of the Institute of Actuaries), and I also hold Microsoft Professional Program certificates in Data Science, Big Data and Artificial Intelligence.  

In any case, a true scientific approach means that anyone, no matter how qualified or unqualified they are, whether adult or child (or in an AI world, a robot or program!) is free to challenge any assertion, no matter how eminent the group that makes that assertion is.The scientific community and everyone else can then see whether the response to a challenge appears sound or not. If not, then the assertion must be corrected, refined, or dismissed. 

You are completely free therefore to discount or listen to what I say as you wish. But I hope this post will be of some interest to those with an interest in trying to understand what (to me, as someone new to the paper, and from my interpretation of it) may be driving current government thinking.

Key observations so far from the paper

The most important points that I take from the paper are:

Assumed death rate

• The model assumes an average death rate of 0.9% for those infected in the UK by Covid-19 (Infection Fatality Rate, see page 5 of the paper). It is an average rate, so will be lower for some (e.g. younger people) and higher for others (e.g. older people or those with significant existing medical conditions)
• It assumes that on average 4.4% of infected people will need hospitalisation, and of these 30% (so 1.32% of infected persons) will require critical care. It assumes that on average about half of the latter will die, despite receiving ICU (Intensive Care Unit) treatment. (Again see page 5). Note that this also means that 3.08% of infected persons are assumed to need non critical hospital care. 
• This isn't stated in the paper (perhaps it should be?), but if we make the assumption that deaths only occur from those who require hospitalisation (which I think should be correct to first order), in order to arrive at the overall death rate of 0.9%, given that we know that half of the 1.32% critical care cases are assumed to die (so 0.66%), it follows that 0.9 - 0.66 = 0.24% out of the 3.08% of non critical care hospitalisations (so about 1 in 13) are assumed to die. The converse is that treatment of critical cases saves 0.66% of those infected, and treatment of non critical cases saves 3.08 - 0.24 = 2.84% of infected cases.
• An important question is what would happen to new infected cases if the NHS was overwhelmed and unable to treat any further COVID-19 patients. From the above, the death rate would go up from 0.9% by 0.66 (from critical cases no longer being able to be saved) + (some but not all of 2.84 non critical cases no longer able to be saved). So the death rate would rise from 0.9% to somewhere between 1.56% and 4.4% of infected cases, depending on how many non critical cases can survive without hospitalisation.

On this basis, the death rate of new infected persons is likely to almost double, and perhaps rise almost five fold if the NHS capacity is breached.

Without intervention, the paper predicts peak infections in about 3 months time (so around mid June) and gives a projected 510,000 deaths.

Given the UK population of about 66 million, that would imply a death rate of 510,000 / 66,000,000 = 0.77% assuming 100% of the population becomes infected, or 0.96% if (as mentioned elsewhere in the paper) only about 80% of the population becomes infected.

Both of these rates are much closer to the 0.9% average rate the paper assumes allowing for treatment, than the 1.56% to 4.4% range that would seem to apply if the NHS were to be overwhelmed.

In fact the paper says on page 7 (the bold emphasis is mine):

we would predict approximately 510,000 deaths in GB and 2.2 million in the US, not accounting for the potential negative effects of health systems being overwhelmed on mortality. 

Given that the paper also states (lower down on page 7) that without intervention the NHS ICU bed capacity would be overwhelmed by the second week in April, the actual number of deaths would be projected (on the model's own assumptions after allowing for no NHS treatment being possible) to be between 884,000 and just under 2.5 million. 

Hence do nothing is not an option

Understandably given the above (since even 510,000 deaths would represent a doubling of the typical yearly number of deaths - from old age, sickness, motor and other accidents etc - in the UK from a population of some 66 million), the modelling team and the government have concluded that doing nothing is not an option.

They therefore look at two broad types of strategies: 

• mitigation, under which the epidemic is allowed to continue to spread, but at a lower rate and not amongst the most vulnerable (older people and those with significant existing medical conditions), and
• suppression, under which the goal is try and stop the virus spreading, until such time as a vaccine or effective treatment can be found.

Mitigation strategy

(The image above is from figure 2 on page 8 of the paper)

The paper states that under the model used, the best practical mitigation strategy that they have been able to suggest (see page 6) is a mixture of case isolation (infected persons or persons with symptoms self isolate for 7 days), home quarantine (for people in the same household as someone with symptoms, for 14 days) and social distancing for those aged over 70 (to protect them from becoming infected). This is the blue (lowest) curve in figure 2 on page 8 of the paper (see screenshot above). The red line is the NHS critical care bed capacity, above which further patients with critical needs can't be treated.

But even under that strategy, although it "reduces peak critical care demand by two-thirds and halves the number of deaths", it would still result in an 8-fold higher peak demand on critical care beds than the NHS has capacity for. Page 16 states (again the bold emphasis is mine):

even if all patients were able to be treated, we predict there would still be in the order of 250,000 deaths in GB.

Note that In the same way that the 510,000 deaths quoted under the "Do nothing" policy is a significant underestimate (because of the much higher mortality rate that applies once the NHS is overwhelmed), the figure of 250,000 also looks to be much lower than it would be in practice.

Not surprisingly, the group and the government have concluded that mitigation is no longer an option:

effectively all COVID-19 cases in figure 2 above the red line become subject to an assumed mortality rate of about 1.5% to 4.4% rather than 0.9%.

Hence the move since the publication of the paper to a suppression strategy (see page 16): 

We therefore conclude that epidemic suppression is the only viable strategy at the current time.

Suppression strategy

Here the goal is to keep the number of infections below the level at which the NHS is overwhelmed. This wasn't possible with mitigation, so the strategy is to stop the infection spreading by severely reducing social contact.

However, the social and economic costs of doing this are considerable (so this is not sustainable beyond a few months), and given that Covid-19 is a global pandemic, the epidemic is very likely to flare up again very soon after restrictions on social contact are released. 

This strategy therefore involves a succession of alternate periods of isolation for the population as a whole, followed by temporary relaxations, with such iterations continuing until either a vaccine has been found, or an effective treatment, or massively increased health service capacity.

The paper states that under the model used, the best practical suppression strategy that they have been able to suggest (see page 11) is:

Combining all four interventions (social distancing of the entire population, case isolation, household quarantine and school and university closure) is predicted to have the largest impact, short of a complete lockdown which additionally prevents people going to work.

(The screenshot above is from figure 3 on page 10 of the paper.) 

The strategy proposed by the group (and it seems adopted for the time being by the government) is shown as the green line. 

Note that it assumes that the current restrictive measures remain in place for about 5 months, and are likely to prevent the disaster scenario of the NHS capacity being overwhelmed, thus keeping the mortality rate at the assumed lower level of 0.9%.

It then assumes that some of the restrictions are temporarily relaxed in about September of this year (with schools and universities reopening and social distancing relaxed), before being reimposed again after what looks like 3 to 4 weeks (before the green line starts rising steeply towards the NHS capacity red line.

We face 12 to 24 months of facing very significant social restrictions for about 2/3 of the time

This approach is shown in the screenshot below of figure 4 from page 12 of the paper:

What this chart implies is that, until either:

• an effective vaccine is found, and sufficient stocks of it become available, or
• health service capacity is very significantly increased, or 
• an effective method of home treatment (not requiring skilled medical equipment or staff) is found, or
• a suitable combination of the above

we in the UK (and probably those in other countries too) face many months (not just the next 5, but probably a year or two) of alternating restrictions and partial relaxations (with social distancing relaxed, and schools/universities reopening), with full restrictions operating an estimated 2/3 of the time.  

The changes to our way of life (and to our economy) are likely to be profound.  But the alternative looks very stark: infected patients facing average death rates of somewhere between 1.5% to 4.4% (the 0.9% that we face even if the health service capacity were a thousand fold bigger is bad enough) and doctors and nurses facing agonising decisions as to who to treat.

Some (possibly significant) queries

The above are the conclusions that I draw from reading the paper. I may be mistaken, and in any case corrections or comments are welcome.

I do have some queries. Some are very minor so if I post them at all it will be in a follow up post.

But two queries might have a significant impact on the severity of the regime that it looks we (the human population) face from the group's analysis (for which I believe they are owed many thanks given the considerable work involved, and its importance):

Can the restrictions be relaxed more often than just a 1/3 of the time?

In figure 4, the peaks are projected to be much lower after the first one.  ICU (Intensive Care Unit) surge capacity is stated (page 13 of the report) to be approximately 5,000 beds.  

Couldn't we switch to a higher trigger level for reimposing restrictions (such that the peaks are around 1,000 or even 2,000 ICU beds rather than the 400 shown in figure 4)? 

This would demand a lower proportion of time in suppression mode than the 2/3 projected in figure 4.

To what extent might the assumed mortality rate be significantly overstated due to not counting infected cases with little or no symptoms?

.One of the paper's key assumptions is the Infection Fatality Rate (the death rate for infected patients) of 0.9%.  That is stated (on page 5) to have been arrived at from an analysis of known deaths out of total known cases in China, and adjusted for the UK's age distribution.

But the UK's Chief Medical Officer has stated that the government's best estimate of the true current number of cases (including patients who are unaware that they have COVID-19, because they haven't been tested) is around 40,000 to 50,000, as compared with the most recent number of confirmed cases so far (2,626 as at 0900 on 18 March 2020). 

Of course it is not clear whether "40 to 50 thousand" means currently active cases, or the total number infected so far, which would be comparable with the 2626 known cases.

But it does seem possible that if we (or the Chinese) could test everyone, we might find that the number of current cases is actually 10 times higher than the current number of confirmed cases. In which case the correct average mortality rate to assume might not be 0.9% but 0.09%.

Because of exponential growth in the number of infections, NHS capacity would still be breached without suppression, but instead of the post breach death rates being somewhere between 1.5% and 4.4%, they would be between 0.15% and 0.44%, leading to extra deaths compared to a normal year of the order of 100,000 to 290,000 - still bad, but better than the current 600,000 we face the prospect of, even under the suppression policy (given that the government has said that most of us are likely to be infected at some point), on the assumption of a death rate of 0.9%. 

I gave a presentation to the Channel Islands Actuarial Association earlier today, and a similar one in January to the Staple Inn Actuarial Society.

I provided copies of the materials I used to those two Societies, but I have had a few requests for a copy from other people too so:

You can find a copy of the slides I used earlier today here. (I have tried to make the slides reasonably self-documenting but if you want further information, please get in touch). 

My talk was about my experience of taking over 30 MOOCs (Massively Open Online Courses), the differences as I see them between data science, big data and AI and the courses which I found most interesting from an actuarial point of view. I also talked on the ethical uses & challenges of AI.

One of the common use cases for an Azure function is to run on a timer to check the status of one or more Azure queues.

You can do this very easily via an Azure v2 function because these integrate very easily with their associated Applications Insights resource. (If you create the function via the Azure portal, you can choose - and I recommend - to create an associated Applications Insights resource at the same time.  Otherwise you can add such a resource after creating the function.)

It turns out (although this is not immediately clear from the documentation) that you don't have to do anything to connect your function to its Application Insights resource: if you want to add a custom metric (e.g. the length of a particular queue), all you have to do is call LogMetric on the logger than is passed to the function.

For example, for a queue called queueName whose approximate length is queueLength, simply call:

log.LogMetric($"Queue length - {queueName}", (double)queueLength);

So with very little code you can track and monitor the approximate length of your Azure queues over time via creating charts of these custom metrics and even add them to a custom Azure dashboard that you can check periodically to try and find out:

• a) what your normal patterns of queue usage are
• b) spot unusual patterns which could be an early warning sign of problems or extra resources being needed.

Set up automated alerts if your queues grow bigger than expected

And you can even set up alerts to send yourself and key colleagues an email if a queue (including a poison queue) has an unexpectedly high number of messages.

To do this: in the Portal, via Monitor, Alerts, New alert rule, then choose the ApplicationInsights cadsqueuemonitorfuncapp1 resource.

In Configure signal logic, choose Metrics as the signal type and AzureApplicationInsights as the service to be monitored, and then you can choose any of the custom metrics (i.e. the length of a particular queue), and add an alert to e.g. send an email to a group of addresses if the queue length exceeds a specified number.

Conclusion

This is a great feature of Azure Functions version 2, and combined with custom metrics in Application Insights, custom dashboards and Alerts can make monitoring your Azure queues much easier!

After all, that is what OneNote is designed for, whereas Word is designed to be a general word processing tool!

If you put your notes in Word, you have to use Windows File Explorer to find them, and organise them, and when you are searching for notes (and content within notes), you have to keep switching between File Explorer and Word.

I have found OneNote much easier both to organise and to find content within notes.  OneNote has an excellent search facility, and you can stay within OneNote to find any of your notes.

OneNote's hierarchy: Notebooks, Section Groups, Sections and Pages

For very different items, you can use different Notebooks (e.g. Personal and Work).

Within a notebook, you can organise notes within different Section Groups, each of which has different Sections, each of which has different Pages.

For example a student revising for exams could have different Section Groups for each subject (e.g. Magical Creatures, Divination, Alchemy), and use different Sections for each major topic within a particular subject (e.g. Gold, Silver within Alchemy).

The instructions and screenshots below are for OneNote for Windows 10 (which is now actually just called OneNote).  The previous version of OneNote was OneNote 2016, which Microsoft are replacing with OneNote for Windows 10.  The steps for OneNote 2016 would follow very similar principles however.

Example of a well structured OneNote notebook for revision

How can you achieve this sort of structure within OneNote? By making use of the hierarchy of items within OneNote as follows.

How to create a new Section Group

In the bottom left of OneNote, you should see + Add section. Right click this and click on New Section Group:

Note: You can rename most items (Section Group, Section, or Page by right clicking them and Rename should be one of the options).   Here is an example of naming or renaming a Page:

So once you have created your new Section Group, give it an appropriate name.

How to create a new Section within a Section Group

Right click the section group in the left navigation pane and click on New Section, and there is a  Create New Tag option (or you can select existing tags):

Give your new Section an appropriate name.

How to create a new Page within a Section

Select (left click) the section in the left navigation pane and click on the Add Page link below and to the right:

Give your new Page an appropriate name.

Once you have organised your notes in this way, how can you find them, and read them?  The left toolbar is very useful for this:

OneNote's left toolbar

This has items to Show/Hide Navigation, Search, and Recent Notes:

If you want to reorder sections, you can drag and drop within the Navigation Links. You can also right click sections or pages and there is a link to move or copy them. 

You can use the search icon (magnifying glass) to search by content, and also by any tags you have marked any of your pages with.

To add a tag

Click on the dropdown arrow to the right of the tags link (a ticked box) in the toolbar:

I hope you find this post useful and that, like me, OneNote enables you to do your work and studying more effectively. 

In particular, not only will this version include Blazor, but it will include support for machine learning.  https://msdn.microsoft.com/en-us/magazine/mt848631.aspx says:

NET Core 3.0 will also fully support ML.NET, our open source machine learning framework built for .NET developers. ML.NET powers products like Azure Machine Learning, Windows Defender and PowerPoint Design Ideas. Using ML.NET you can add many popular machine learning scenarios to your apps like sentiment analysis, recommendations, forecasting, image classification and more. Learn more at bit.ly/2OLRGRQ.

This was my 9th course on the professional program, and this month (July 2019) I am taking the final course, a capstone project designed to test a lot of the material learned during the previous courses.  I have had a quick look at this (I couldn't until now because the capstone courses can only be taken in the first month of every calendar quarter), and it is an interesting project: developing an algorithm to successfully classify sound files as to whether the accent used is Canadian, English or Indian! So I am looking forward to doing that.

Once I have finished the AI professional program, I will write up an overview of it (as I did here with the Data Science and here with the Big Data professional programs), but here are some brief comments about the Reinforcement Learning Explained course.

Arguably this is the most actuarial/mathematical of all the courses I have done so far, whether on data science, big data or AI. It involves optimisation in a stochastic process, and includes dynamic programming and the Bellman equations, which I remember doing at university a few decades ago. Quite a lot of python programming too.

Reinforcement Learning includes algorithms similar to the ones used by Alpha Go Zero (the program that managed to beat Alpha Go, the program that had beaten the human world champion at Go).  Alpha Go taught itself initially by studying many previous games played by humans. Alpha Go Zero was entirely self taught: i.e. all it started off with was the rules of Go, and it learned by playing itself and gradually improving!

The image at the top of this article is of a much simpler problem than Go, a Simple Room Environment in which the objective is for an agent (think of this as a player in a game) to learn how to reach the bottom right square as quickly as possible starting from the top left square.  There are walls in place which block certain routes. The agent is represented by the yellow square.

The image shows the progress of an agent using a SARSA (State Action Reward State Action) algorithm as it learns during 12 stochastic episodes (plays of the game).  The game stops after 50 moves or if the goal (the bottom right cell) is reached before then.  The Length in the charts is the number of moves (with 50 normally denoting failure to reach the goal), and the Reward is the total reward received during the episode (a score of 1 means that the goal was reached). 

You can see from the charts that towards the end of the 12 episodes, the agent seems to have learnt to reach the goal consistently (i.e. it will no longer fail).  This is not necessarily going to be the case however, nor would this necessarily be optimal: the agent may have found a lengthy route to reach the goal, so often in Reinforcement Learning problems, it is necessary to continue exploring from time to time (e.g. under an epsilon greedy process, see https://en.wikipedia.org/wiki/Multi-armed_bandit) in the hope of finding an even better route.

PS I have complained to edX about the course having been advertised as only taking 24 to 36 weeks yet containing a lot more material and lab work than previous courses. And about other problems too: very poor English in the material and questions, very poor pronunciation in some of the course videos (and hence the video transcripts, which clearly were never checked by edX), no responses from edX to discussion forum posts after 2 months! I hope edX will sort some or all of these problems out in future versions of the course, but be forewarned that it may take considerably longer than other courses in case they have not.

I look forward to helping the Institute and Faculty of Actuaries, our members all around the world, and actuaries generally, to play the very significant part that I believe we can in helping businesses, organisations and individuals to make the most from artificial intelligence and big data, in an ethical way, in all types of business areas.  The IFoA's forthcoming new Certificate in Data Science (announced at this week's AGM by incoming President John Taylor) is a very exciting new development and I am looking forward to finding out more as I rejoin Council.

This course was all about image segmentation (the ability for a computer to recognise the existence, location and class of different types of objects in an image, e.g. cars, buses, lane markings and pedestrians in an autonomous car application) and deep segmentation, the foundation of scene understanding (the ability of a computer to understand to what object in an image every pixel in the image belongs).

It seems that since about 2012, deep learning models (AI neural networks with many layers and parameters) have been outperforming traditional methods requiring human intervention (such as Viola Jones and Histogram Oriented Gradient also known as HOG) in vision processing problems.

Another major incremental step in vision processing is the process of transfer learning, where a deep learning model trained on one task (e.g. recognising one set of objects) can be reused in the same or similar problem domain on a different set of objects.  This is because the earlier layers of the model normally enable it to recognise general features of objects (such as edges, corners) and it only needs to learn how to amend its final layers to recognise the specific features of the new set of objects. 

The course required using Python and the computer vision part of the Microsoft Cognitive Toolkit (CNTK) in Jupyter notebooks in Microsoft Azure.

Why is DevOps important? Huge increases in efficiency for tech companies

#DevOps is the art and science of trying to automate/streamline as much as possible many stages (i.e. humans are still involved in the writing of code) of producing and deploying software.

A formal definition is:

DevOps is the union of people, process, and products to enable continuous delivery of value to our end users. You cannot buy DevOps and install it. DevOps is not just automation or infrastructure as code. DevOps is people following a process enabled by products to deliver value to end users. (Donovan Brown, Microsoft DevOps Program Manager).

DevOps involves things like running automated tests before code is deployed, storing infrastructure and compliance requirements as code (thus increasing reproducibility when having to deploy to different environments/machines, and increasing confidence by regulators that organisations are less likely to break compliance via human error). 

Just tech companies? All companies will need to transition into tech companies to take advantage of big data and AI

With the advent of big data and artificial intelligence, experts like Bernard Marr say that it is essential for companies in every industry to transition into technology companies if they want to stay ahead of the pack. Adopting a DevOps approach for technology companies can lead to huge reductions in flexibility and hence time to market.  According to a 2017 report by Puppet Labs  (now called just Puppet), high-performing I T organisations experience three times lower change failure rates and recover 24 times faster than their lower-performing peers.  They also deploy 200 times more frequently.

Note: if you are contemplating doing this course, please note that (as often happens with edX courses) the instructions for the labs (exercises) seem to be quite a bit out of date, since both Microsoft's Azure platform and Visual Studio software have advanced quite a bit since they seem to have been written.

I look forward to working with the rest of the Committee and the members of the Society over the next few years, particularly to try and promote Bournemouth and the Wessex region as a major centre of excellence for insurance and data science.

Overview

The program involved taking 10 courses which were:

1. Microsoft Professional Orientation: Big Data (completed in October 2017)
2. Analyse and Visualize Data (completed on the Data Science program in July 2017)
3. Work with NoSQL Data (August 2018)
4. Query Relational Data (completed on the Data Science program in June 2017)
5. Implement a Data Warehouse (September 2018)
6. Process Big Data At Rest (December 2018)
7. Process Big Data In Motion (February 2019)
8. Orchestrate Big Data Workflows (February 2019)
9. Build Big Data Analysis Solutions (completed on the Data Science program in March 2018)
10. Big Data Capstone Project  (April 2019)

For several of the courses, alternative options were available (e.g. between Azure Data Lake Analytics and Hadoop for course 6, and between Azure Stream Analytics and Azure HDInsight for course 7). I chose the first option in each case to get to know the pure Microsoft technology because I felt that this would be more consistent and integrate more easily with the other Microsoft tools in their Azure cloud offering.

Note that although I started the program in October 2017 I didn't complete it until April 2019.  That was because during May 2017 to April 2018 I was concentrating on completing Microsoft's Professional Program for Data Science (for my Data Science journey please see here).  As noted above there was some overlap: three courses from the latter were also part of the Big Data program.  

It took me 234 hours of part-time study to complete the Big Data program, 146 hours excluding the 3 courses that I had already completed as part of the Data Science program. My average mark across the 10 courses was 97.7%. (See here for some tips about how to achieve high marks in edX and other online courses).

Caution: for some of the labs (online exercises) the cloud computing resources can be quite expensive (£10 or significantly more if you leave the resources idle for several days, as compared to $99 for each course), so I often waited until the end of the course before doing the labs (and then deleted the relevant cloud resources as soon as I had finished the labs). 

What is Big Data?

Precise definitions vary, but for this course "Big Data" was defined as a form of data with either a high volume (more than a million or two of records), or a large variety of formats (e.g. images, videos), or which comes very quickly (with high velocity).

During the course we explored the management and analysis of several different types of big data, including:

• structured data (with a fixed schema - a fixed list of fields - and fixed relationships between different tables or sets of data)
• unstructured data (with a variable schema - a variable list of fields - or where the data could be very general, e.g. tweets or blog posts or other documents)
• data at rest, where new records are added at regular intervals, e.g. once a day or once a week
• data in motion, where new records are added at very frequent intervals, e.g. in real time like stock market prices, or streaming data from Internet of Things ( IoT) devices.

The final course was a capstone project which involved three separate challenges that tested my ability to build Big Data solutions for:

• processing data at rest
• processing data in motion, and
• orchestrating a Big Data solution.

I try to give a brief idea below of what was involved in each course in the program. Note: like many other online courses, the material is updated from time to time, so may have changed if you take the course at a later date.

Course 1: Big Data Orientation

This covered

• an introduction to data, data formats and data files, storing files in Azure (Azure Storage and Azure Data Lake Store)
• an introduction to relational databases including data warehouses
• in introduction to NoSQL databases (key-value, document and graph databases).  SQL stands for Structured Query Language, and NoSQL can either stand for "No SQL" or "Not only SQL" and is useful for unstructured or semi-structured data such as blog posts or tweets
• an introduction to big data processing (what is big data, data at rest, data in motion, batch and real-time processing in Azure, batch and real-time processing in HDInsight)

Course 2: Analysing and Visualising Data with Power BI

I completed this as part of the Data Science program.  This showed how to produce visuals (and perform some analysis) in Microsoft Power BI (Business Intelligence).

Microsoft Power BI (like other programs e.g. Tableau) is very useful for producing interactive dashboards for senior management and others to extract insights from data.  You can click on one chart to highlight a particular feature or filter the data, and all the other charts or tables adjust accordingly.

Course 3: Work with NoSQL Data

NoSQL also known as "not only SQL" is often used for unstructured data (or at least data that is less structured than relational data, and for which the fields can vary for different types of records within the same table).

This covered an introduction to:

• Key-Value stores, Azure storage tables
• Azure Cosmos DB
• MongoDB
• More NOSQL database solutions (including Cassandra, Lucene & Solr, Azure search, HBase and Redis)

Course 4: Querying with Transact-SQL

This was my second course from the Data Science Program and was also one of the courses on the Big Data Program.  This taught how to query relational data to extract meaningful information.

This screenshot shows one way of extracting related data from two different tables:

This screenshot shows part of the course describing Stored Procedures (bits of code that run on the database server, which is often more efficient than extracting the relevant data and doing the processing on a different server, e.g. a web server): 

Course 5: Delivering a Data Warehouse in the Cloud

My 5th course was Delivering a Data Warehouse in the Cloud. This gave an introduction to SQL Data Warehouses, a specialised type of relational database where MPP (Massively Parallel Processing) can be used to query large volumes (hundreds of millions of records or more) of key data, which has usually been extracted from even larger volumes of raw data stored elsewhere.

Topics covered included:

• Designing and querying data warehouses (distributed tables, replicated tables, partitioned tables, indices and statistics, monitoring queries)
• Integrating data (loading data from raw data sources, including via Azure Data Factory, Azure Stream Analytics and Polybase).
• Analyzing data with Azure Machine Learning, visualizing data with Power BI, and integrating with Amazon Athena and Redshift
• Managing data warehouses (including columnstore fundamentals, SQL data warehouse security, managing computations including analyzing workloads, and backups).

This screenshot shows the architecture of Azure SQL Data Warehouse, with Massively Parallel Processing:

Course 6: Process Big Data at Rest (with Azure Data Lake Analytics)

Course 6: Process Big Data at Rest (with Azure Data Lake Analytics). Topics covered included:

• an overview of Azure Data Lake, an introduction to U-SQL, U-SQL fundamentals (U-SQL is a language that allows you to combine SQL with C#)
• Using a U-SQL Catalog
• using C# functions in U-SQL
• monitoring and optimizing U-SQL jobs

Course 7: Process Big Data in Motion

My 7th course was Process Big Data in Motion, Processing Real-Time Data Streams in Azure. This gave an overview of how to process large volumes of streaming data, e.g. from IoT (Internet of Things) devices in real time. Topics covered included:

• ingesting real-time data with Azure Event Hubs
• Ingesting real-time data with Azure IoT Hubs
• Analysing real-time data using Azure Stream Analytics
• Working with Temporal Windows

Course 8: Orchestrating Big Data Solutions

Parallel or distributed processing is often required when doing big data extraction, transformation and analysis.  Some means of orchestrating the different tasks (including checking that they have all run successfully, rerunning those that failed, alerting the user if reruns have been unsuccessful, doing the various steps in the right order, combining the output from different tasks etc) is needed. This course covered how to use Microsoft's solution to this problem, Azure Data Factory.

Topics covered included:

• Introduction to Azure Data Factory
• Pipelines
• Scheduling Pipelines
• Transformations

This screenshot shows Azure Data Factory's activity window explorer, which shows tasks that have completed successfully in green, those for which there is no data yet in grey, and those for which  there is data but which haven't yet started in orange:

Processes which have run but failed are shown in red, as in this screenshot:

Course 9: Analyzing Big Data with R

My 9th course (which I completed previously as part of the Microsoft Professional Program for Data Science) was Analyzing Big Data with Microsoft R Server. Standard R is limited to the memory of the computer running it. This showed how to use Microsoft R server for larger datasets.

Screenshot showing an example of some R server code:

A screenshot showing some analysis of New York taxi data:

Course 10: Capstone Big Data

The 10th and final course (completed in April 2019) was a capstone project (one designed to bring several earlier courses together) with three separate challenges that tested my ability to build Big Data solutions for:

• processing data at rest
• processing data in motion, and
• orchestrating Big Data solutions.

Challenge one: processing big data at rest

This involved analysing data from about 16,000 text files, representing weekly stock inventory data during a full year for a large retail organisation with many stores. I had to extract the data, process it and send the processed data for each week to a data warehouse. Challenge one involved working on a particular week's data.

Challenge two: processing big data in motion

This involved receiving real-time stock adjustments from a simulated chain of retail stores, aggregating it over non overlapping one-minute intervals and analysing the results.

Challenge three: orchestrating a Big Data solution

This involved processing the weekly inventory data from challenge one by using Azure Data Factory to load data for each store and each week for a full year into a data warehouse and then analysing the output.

Conclusions

Storing and analysing big data (data of either large volume - e.g. more than a couple of million records, variety or velocity) requires quite different techniques from those normally used for smaller data sets. NoSQL databases can be very useful to handle unstructured data, or situations where the data schema may change in future. Different techniques are needed for handling real-time (streaming) data as compared to data at rest. In many big data situations, parallel or distributed processing is needed, together with some means (e.g. Azure Data Factory) to orchestrate (i.e. manage and combine) the different processes. 

I found this professional program a very useful complement to the Data Science program (which covered predictive modelling itself, rather than how to use the very different techniques required to process big data). 

In particular there will often be far too many carriage returns, leading to a large number of rather narrow lines.

You can improve the formatting by hand by going through and removing the unnecessary carriage returns, but this is tedious and time consuming.

I wrote a small C# WPF (Windows Presentation Foundation) program to do this for me. You can download the code  for this (for use free of charge, at your own risk, using say Microsoft Visual Studio) from this Github repository. (Github is a place where software developers can share and collaborate on code).

To use this program

In the edX course, click on the .txt file link for the video. This will normally put the transcript file (e.g. VideoTranscript.txt) in your downloads folder.

Run the WpfTextEditor project (in debug mode will do). The following window should appear:

Click the Run button.

Use the common dialog box that appears to navigate to the .txt file that you wish to reformat the contents of more neatly (e.g.Downloads\VideoTranscript.txt).

Select the desired transcript file (e.g. VideoTranscript.txt) file and click the Open button.

The program should read in the contents of the text file, reformat it more neatly and put the results into the Output text box of the program's window:

Click somewhere inside the Output text box then (on a Windows machine) press Ctrl+A (to select all the text):

Then type Ctrl+X (to cut all the text). You should then be able to paste (Ctrl+V) the formatted video transcript text into the relevant place (e.g. the relevant part of your OneNote page for the module containing the video).

Conclusion

This should make your notes of the videos in the course more readable. I also recommend taking screenshots of key parts of the videos and pasting them at the relevant places in your transcript notes.

I've not managed to quite achieve this, my average mark on the courses is about 97%, for reasons which I will set out below.

For anyone interested, here are a few tips from my experience so far on how to maximise your marks on edX courses (and I think they should also be useful for other online courses):

Take comprehensive notes (I used Microsoft OneNote, one page for each module)

For each module in an edX course, I usually copy most of the text from each edX page and paste it into OneNote page.  I also do this for most of the transcripts from the videos, and I take screenshots of key parts of videos and paste them into the same OneNote page. 

Here is an screenshot showing the start of a OneNote page I used to take notes of a module in the "Delivering a Data Warehouse in the cloud" course:

I also take copies of the assessment questions, and keep a record of my answers to the questions. (NB the notes you take must be for your personal use and not to be shared with others, in order to comply with the edX Honour Code and terms of use).

Why are comprehensive notes useful?

1. When answering the assessment questions, you can use OneNote's very powerful search facility to search (usually within the same page using Ctrl+F, since the questions for a module should relate to that module, but otherwise Ctrl + E to search across all your notebooks) to find relevant parts of the course. If you haven't taken notes, you will need to either be able to remember all the material (not always possible, as there is a lot), or search back through the pages and videos on edX.
2. The search facility in OneNote is also very useful when doing the labs (exercises) and the final exam/assessment at the end of the course. (Like the real world of work, edX exams are open book, so it is perfectly fine to search your notes - or even the internet - and that's fine, because that is what people do day in and day out in their jobs.  What matters is getting the right answer, providing of course you are not cheating by getting someone else to do the questions for you.)
3. You are likely to need some of the material months or even years after you have finished the course, and it may not longer be available on the edX site. Many times I have needed to do something in real life work and have been able to refer to my notes to save me (and colleagues) a lot of time.

Incidentally, I recommend using OneNote for many other aspects of your work, not just taking notes from online courses.  I find it very useful to store things like: problems and how I managed to solve them, how I set up things in Azure (so that the next time I do something similar I can use that as a template), notes of meetings etc.

Whenever possible, check your answers in practice before submitting them

If the assessment or exam question involves writing code (whether it be T-SQL or USQL for database queries, or HTML or CSS or javascript for web pages, or R/Python/C# for machine learning, or PowerBI/Tableau/Excel for data visualisation), you will often have the opportunity to try out your proposed answer in one or more tools (e.g. SSMS [Sql Server Management Service] for T-SQL, Visual Studio Code/Visual Studio/CodePen or even just Notepad or other basic text editor and a browser for HTML/CSS/javascript, Azure Machine Learning Studio or Jupyter Notebooks for R/Python/C# snippets, Visual Studio Code or Visual Studio 2019 for C# code). 

Use your problem solving skills: if the suggested method isn't working for you, try another, and another!

Sometimes the edX instructions for online labs (exercises) are out of date, so you can't follow them exactly.  Or there might be some other reason why the suggested approach isn't working for you.  Well, just as you would do in your real life work, you will often be able to think of another way to attack the problem. 

For example, instead of analysing data using T-SQL, you might be able to do it using PowerBI, Tableau, or just Excel!  (Even if the data won't fit into an Excel worksheet, because there are more records than Excel's current maximum number of rows of just over a million,  you can use Power Pivot and Excel's Data Model, together with Pivot Tables and charts to analyse it).

What went wrong, why didn't I get 100%?

For maths/IT questions, I believe that with diligent study one should usually be able to get 100%, assuming the course is well taught. (This is in contrast to business questions, where often there is no right answer - or more than one, and arts/humanities questions, where again many different answers may be possible and subjectivity is usually unavoidable in the marking process).

So I am always disappointed not to get 100% on these edX courses, and that has happened on several courses, which is why my average mark on the courses I have taken so far is about 97%.

So what went wrong?

• In some cases this was human error on my part: I made a mistake (in particular, some of the final exams are timed, and I either didn't have time to check my answer in practice using one of the methods suggested above, or to find the relevant notes in my OneNote pages, or my notes were not sufficiently comprehensive)
• In some other cases it was human error on the part of edX: in some cases the edX course material is out of date (e.g. has not reflected changes to the Microsoft Azure portal within the last 6 months or more), or the questions are ambiguous (on some courses the material has many spelling mistakes, so other errors are not surprising), or the questions are on material that has not been covered in the course
• it can be much harder to achieve over 90% on the Capstone courses (the final ones in each Professional Program) because - quite rightly - the projects are much more open ended with much less guidance, so as to test that you are able to perform relevant work without being guided step by step through the process. And for a machine learning exercise, there will usually not be a single "right" predictive model to use.

I hope you have found this post useful.  As ever, I welcome your feedback!

When done accurately and before any clever processing, the number of cashflows produced can easily run into the millions, and for a large pension plan, into the billions. (When I have time, I plan to explain why, and talk about some of the clever processing that can be done).

This situation is one that obviously calls for parallel or distributed processing, with the work shared out between many machines.

In this post, I explain why so many cashflows are produced when calculations are done accurately.

Example UK defined benefit pension plan

Let's consider an example UK defined benefit pension plan, with roughly equal numbers of active, deferred pensioners and pensioners. (Individual pension plans will differ significantly from this, but this simplistic mix is not far from reality for several large pension plans). Let's assume that the pensioners are split in the ratio 20:14 between pensioners and partners (widows/widowers) - again simplistic but not unreasonable/atypical.

For round numbers, let's assume 33% active membership, 33% deferred pensioners (dps), 20% pensioners and 14% partners. For illustration, let's also assume that the average age of the actives is 37, those of the dps is 48, the pensioners 69 and the partners 73.  For simplicity, we will take members of these ages as representative model points for each class, and later on we will apply the 33%/33%/20%/14% weightings to determine the expected total number of cashflows. (The true calculation adds up the cashflows from each member in each membership category, but for ease of demonstration the approach taken in this article will suffice to illustrate the point. We will also see that the average age assumed for categories other than active members is of relatively little importance: it is the actives who produce the vast bulk of the cashflows). 

We will also assume for simplicity that pension plan members have a partner who is 3 years younger than them. (Again, in practice this will vary, but this is not a material factor in determining the scale of the number of cashflows required). 

Let's also assume that the pension plan's normal retirement age is 65, and that on average, members have two different slices of benefits (i.e. their benefits are split into two parts, each of which - due to changes in plan rules or legislation - has different pension increases or other behaviour).  Let's also assume that one of those slices includes some GMPs (Guaranteed Minimum Pensions), and that the terminal age of the mortality tables used is 120, i.e. no member or their partner is assumed to live beyond age 120).

How many cashflows should we expect from each of our representative members?

385 cashflows arising from the partner (widow/widower) member

The simplest case is that of the partner.  Here for each benefit slice, payments will continue to be made from the current age, 73, to the terminal age, 120.  So for the slice without GMPs there should be 120 - 73 = 47 years of payments. In reality there would be 47 * 12 = 564 monthly payments,  probably 570 payments including the final six during age 120 (death occurs after reaching that age and to first order is assumed to occur half way through the year).  But it is common in pension plan projections to use annual payments.   (If we don't then the number of cashflows is further increased beyond the already large numbers shown in this post). 

To allow for discounting and pension increases during the year, the 12 monthly payments can be approximated very well by two payments, one at the beginning and one at the end of each projection year.

That leads to 47 * 2 + 1 = 95 cashflow payments (including the very last one, paid on reaching age 120.  

That is for the benefit slice without GMPs.  For the slice including GMPs, there will be 3 times as many payments (excess over GMP, Pre 1988 GMP, and Post 1988 GMP).  (Technically speaking, for a female partner, there would have been no pre 1988 GMP, but we will ignore this for the purposes of this post, and besides the recent ruling on GMP equalisation may change this retrospectively!).

So we have 95 * (1 + 3) = 385 cashflows for the partner.

12k cashflows arising from the pensioner member

This is slightly more complicated, because as well as the cashflows being paid to the pensioner, we need to allow for cashflows being paid each year (and subsequent years) to a new partner member of the plan on death of the pensioner during each projection year. (For simplicity I am ignoring here and below the very small number of cashflows arising from payment of a lump sum to a deferred pensioner or active on retirement, and also on the death of a pensioner under a pensions guarantee period, normally  of up to 5 years).

Given that the age of the pensioner is 69, there will be (following the approach shown above for the partner), there will be 120 - 69 = 51 years of payments, leading to 51 * 2 + 1 = 103 payments allowing for a final payment on reaching age 120. That would be for the benefits slice with no GMP.  After allowing for the slice with a GMP, in total there will be 103 * (1 + 3) = 412 cashflows paid to the pensioner.

But for each projection year (from age 69 to 120, including the year following on from age 120 to 121), there is a non zero probability that the pensioner will die during the year, so 120 - 69 + 1 = 52 occasions on which partner payments start being paid. (Of course these partners are normally assumed to be all the same person and that is one of the bits of processing that can be done to cut down the number of cashflows, but for the moment we are talking about the number of raw cashflows that need to be produced).  Each of these will have two benefit slices, one of which includes GMP. 

Analogously to the approach shown above for the partner aged 73, the partner arising on death of the pensioner during the year of age 69 to 70 will be aged 69 - 3 = 66 (66.5 in practice but I am ignoring  the half year for simplicity in this post) and will lead to 120 - 66 = 54 years of payments, leading to 54 *2 + 1= 109 cashflows. The partner arising on death during the year of age 70 to 71 will lead to 120 - 67 + 1 = 53 years of payments, leading to 53 * 2 + 1 = 107 cashflows, etc, right up to the partner arising on death during the year of age 120 to 121 leading to 120 - 117 = 3 years of payments with 3 * 2 + 1 = 7 cashflows.  

This is a total of 109 + 107 + 105 + .... + 7 = 3,016 cashflows. 

Allowing for the two slices, one of which has GMPs, this means that the total number of cashflows paid to partners is 3016 * (1 + 3) = 12,064.

Overall, the number of cashflows arising from the pensioner member is 412 + 12064 = 12,476.

23k cashflows arising from the deferred pensioner member

This is slightly more complicated still, because as well as the cashflows that will be paid on (and following) retirement to the deferred pensioner (and to a partner on death after retirement), we need to allow for cashflows being paid each year (and subsequent years) to a new partner member of the plan on death of the deferred pensioner before retirement. (We could also allow for potential early retirement of the deferred pensioner on ill health, but unlike for active members, where the benefits on ill health retirement can be more generous and hence more financially significant, this complication has been commonly ignored - because leading to a slight overstatement of the liability, hence a slight element of prudence - in the valuation of deferred pensioners.  So I ignore this here for simplicity).

Analogously to the approach taken above for the pensioner member, but applied to a pensioner aged 65 (the normal retirement age), that pensioner will lead to 

(120 - 65) * 2 + 1 = 111 payments for the slice without GMP, and a total of 111 * (1 + 3) = 444 cashflows paid to the pensioner. 

Plus, on death of the pensioner after retirement, starting with 120 - 62 = 58 years of payment or 58 * 2 + 1 = 117 cashflows: 

117 + 115 + 113 + ... + 7 = 3,472 cashflows paid to the partner for the slice with no GMPs, and a total of 3472 * (1 + 3) = 13,888 cashflows paid to a partner arising from death after retirement.

Analogously, for death  before retirement (during the period of years of age 48-49, 49-50 up to 64-65), starting with (120 - 45) * 2 + 1 = 151, and ending with (120 - 61) * 2 + 1 = 119:

151 + 149 + .... + 119 = 2,295 cashflows paid to the partner for the slice with no GMPs, and a total of 2295 * 4 = 9,180 cashflows paid to a partner arising from death before retirement.

Overall the number of cashflows arising from the deferred pensioner member is 444 + 13,888 + 9,180 = 23,512.

Over half a million cashflows arising from the active member!

This is considerably more complicated because, before normal retirement age 65, we also need to allow for leaving service, ill health retirement and death in service.

Exactly analogously to the approach taken for the deferred pensioner member, there will be a total of 444 + 13,888 = 14,332 cashflows arising from normal retirement at age 65.

But in addition, there are huge numbers of cashflows arising during each of the years of age 37 to 38, 38 to 39, ..., 64 to 65  from possible:

• leaving service: each deferred pensioner created will generate several thousand cashflow records (some 23,000 on leaving service at age 48 as you can see from the dp case above, more at younger ages)
• retirement on ill health: each ill health pensioner created will generate fewer but still several thousand cashflow records (in each case more than the 14,332 generated at age 65 on normal retirement, because the member is younger)  
• death in service: each partner created will generate over a hundred cashflow records

The calculations are clearly very tedious to do by hand (or with Excel), so I used software (InQA Limited's Pensions Concerto valuation and cashflow projection tool).

Overall the number of cashflows arising from the active member (this time including the small number of extra cashflows arising from lump sums on retirement, and payment of pension guarantees on death) was over half a million at 646,228. 

How many raw cashflows need to be produced for a medium plan? for a very large plan?

For a thousand member pension plan, with the membership weighted in the stated proportions (33% actives, 33% deferred pensioners, 20% pensioners and 14 % partners), and using the above members as broadly representative, the number of cashflows produced is:

330 * 646,228 + 330 * 23,512 + 200 * 12,476 + 140 * 385 = 223,563,300 or 224 million.

As mentioned before, the actives clearly produce the vast bulk of the cashflows (95% in this case), so it matters relatively little what average age is assumed for the other membership categories.

For a million member plan, the number of cashflows produced would be 224 billion. 

This is not all.  For a continuing pension plan, it is usually necessary to calculate the contribution rate required to fund future accrual, both during a short term period (e.g. the next year), and during the future working lifetime of current active members. In order to do this, three sets of cashflows need to be calculated for active members: the above one (for past service, based on benefits accrued at the valuation date), the benefits that will be produced allowing for accrual during the short term period (e.g. the next year), and the benefits that will be produced allowing for accrual during the future working lifetime.

These are the raw cashflows.  Can we do anything to speed up the processing?

The above are the cashflows that need to be produced at some point during the calculations, in order to do things accurately before any smart processing, and if you want to be able to value each member individually. This can be done with distributed processing and big data treatment/storage.

Of course instead of performing the calculations on individual members, you could perform the calculations on grouped data. If you do this, it is important to make sure the members grouped together are homogeneous with regard not only to demographic factors (age, gender etc), but also with regard to the nature of their benefit slices.  But of course by doing this, you lose the ability to produce accurate member by member values, split by benefit type, slice and decrement.  ("Decrement" here means the mode of exit, e.g. death, leaving service, retirement etc).  

Instead, the approach I prefer is to continue to produce the raw cashflows, and to keep them separately for each member, but to combine them as far as possible (again for each member) without losing the information about benefit type and nature.  This means not only that accurate values split for each member by benefit are produced, but information for accurate risk management (for stochastic modelling or for cashflow matching by nature and type) is preserved.

One example is that for a given member, benefit type, slice and decrement, cashflows at the end of year n and the start of year n+1  can be combined without loss of accuracy, because they have the exact same characteristics including the date of payment.

With this and other smart processing, the number of cashflows that need to be retained can be reduced by a factor of 100 or even more.

The main skill-set I bring to the table is extensive experience of IT including cloud technology and (increasingly) big data/data science/artificial intelligence. I believe there is still much to do to help our members remain relevant in a world that is likely to be ever more dominated by data analytics and AI. With my background I feel well placed to help. 

I wanted to add one to this site (which is a standard ASP.NET Core 2.2 webapp, but modified to use ButterCMS as a blog backend) and after checking with the team at ButterCMS that a facility to do this didn't already exist, I decided to have a go at creating one.  This is the result.

A word cloud is a useful rough visualisation of the relative frequency of particular words, either within an article, or in the case of a Tag Cloud for a blog, how many articles a particular tag has been added to.  Tags which are more frequently used are shown in larger font than tags which are less frequently used, and each word in the cloud has a hyperlink taking you to the articles related to that tag.

Here's how relative frequencies can be calculated, and then saved to an Azure Table using the existing ButterCMS API (Application Programming Interface) methods (perhaps in future ButterCMS might add APIs to produce this information more directly):

Azure Function (to retrieve the necessary information and update an Azure Storage Table with the latest information once a day)

// Date        Who Comments 
// 04 May 2019 PJL Created to cache the information needed for Tag Clouds and Category Clouds from a ButterCMS blog (to avoid slowing down the website)
//                 At the moment, this function updates the cache table daily
// 04 May 2019 PJL Version 1.0 of FunctionApp1.UpdateTables
// **************************************************************

using ButterCMS;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage; // added for CloudStorageAccount
using Microsoft.WindowsAzure.Storage.Table; // added for Table
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace FunctionApp1
{
    public static class UpdateTables
    {
        [FunctionName("UpdateTables")]
        // NB [TimerTrigger("0 0 0 */1 * *")] means every 1 day
        public async static Task Run([TimerTrigger("0 0 0 */1 * *")]TimerInfo myTimer, ILogger log)
        {
            var apiToken = GetEnvironmentVariable("ButterCMSApiKey");
            var client = new ButterCMSClient(apiToken);
            var tagCloudRecords = new List<WordCloudRecord>();
            var response = await client.ListTagsAsync(false);
            var tags = response;
            foreach (var item in tags)
            {
                var page = 1;
                var postsResponse = await client.ListPostsAsync(page, 10, true, null, null, item.Slug);
                var posts = postsResponse.Data;
                tagCloudRecords.Add(new WordCloudRecord { PartitionKey = "Tag", RowKey = item.Name, count = posts.Count(), slug = item.Slug });
            }
            SetRelativeFrequenciesOfAListOfCloudRecords(tagCloudRecords);

            var categoryCloudRecords = new List<WordCloudRecord>();
            var response2 = await client.ListCategoriesAsync(false);
            var categories = response2;
            foreach (var item in categories)
            {
                var itemResponse = await client.RetrieveCategoryAsync(item.Slug, false);
                var page = 1;
                var postsResponse = await client.ListPostsAsync(page, 10, true, null, item.Slug);
                var posts = postsResponse.Data;
                categoryCloudRecords.Add(new WordCloudRecord { PartitionKey = "Category", RowKey = item.Name, count = posts.Count(), slug = item.Slug });
            }
            SetRelativeFrequenciesOfAListOfCloudRecords(categoryCloudRecords);

            var connectionString = GetEnvironmentVariable("NameForYourAzureStorageAccountSetting");
            var storageAccount = CloudStorageAccount.Parse(connectionString);
            var tableClient = storageAccount.CreateCloudTableClient();
            var tableName = "NameOfYourWordCloudsTable";
            var table = tableClient.GetTableReference(tableName);

            var types = new List<string> { "Tag", "Category" };
            foreach (var type in types)
                await DeleteRecordsOfThisType(table, type);

            // add the new records to the table for each type. NB each record in a batch operation must have the same partition key
            await AddBatchOfRecordsToTable(tagCloudRecords, table);
            await AddBatchOfRecordsToTable(categoryCloudRecords, table);

            log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
        }

        private static void SetRelativeFrequenciesOfAListOfCloudRecords(List<WordCloudRecord> records)
        {
            int totalNumber = (from i in records
                               select i.count).Sum();
            foreach (var item in records)
            {
                if (totalNumber != 0)
                    item.frequency = (double)item.count / (double)totalNumber;
                else
                    item.frequency = 0;
            }
        }

        private static async Task DeleteRecordsOfThisType(CloudTable table, string type)
        {
            TableQuery<WordCloudRecord> query = new TableQuery<WordCloudRecord>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, type));
            TableContinuationToken token = null;
            do
            {
                TableQuerySegment<WordCloudRecord> resultSegment = await table.ExecuteQuerySegmentedAsync(query, token);
                token = resultSegment.ContinuationToken;
                foreach (WordCloudRecord entity1 in resultSegment.Results)
                {
                    TableOperation deleteOperation = TableOperation.Delete(entity1);
                    // Execute the operation.
                    await table.ExecuteAsync(deleteOperation);
                }
            } while (token != null);
        }

        private static async Task AddBatchOfRecordsToTable(List<WordCloudRecord> records, CloudTable table)
        {
            TableBatchOperation batchOperation = new TableBatchOperation();
            foreach (WordCloudRecord item in records)
                batchOperation.Insert(item);
            // Execute the batch operation.
            await table.ExecuteBatchAsync(batchOperation);
        }

        private static string GetEnvironmentVariable(string name)
        {
            return
                Environment.GetEnvironmentVariable(name, EnvironmentVariableTarget.Process);
        }
    }

    public class WordCloudRecord : TableEntity
    {
        public WordCloudRecord(string type, string name)
        {
            this.PartitionKey = type;
            this.RowKey = name;
        }

        public WordCloudRecord() { }

        public int count { get; set; }
        public double frequency { get; set; }
        public string slug { get; set; }
    }
}

What the above code does is (within an Azure Function version 2 triggered by a timer once a day):

1. Use the ButterCMS API to retrieve a list of Tags, and for each tag extract the number of blog posts which are tagged with that tag.
2. For each tag, it populates a WordCloudRecord (a record to be stored in the Azure Table) with all the necessary fields except the (relative) frequency.
3. It then calculates the relative frequency for each tag and adds this to each record in the list of tag records (tagCloudRecords).
4. It does the same for Categories, to produce a list of category records (categoryCloudRecords).

At this point, the code could be used within the Home and or Blog controllers of the website but this would mean that the calls in steps 1 to 4 will be made on each page that you want to display the tag or category clouds on, which could significantly slow down your site's responsiveness. 

So instead, the above Azure Function runs once a day to extract the information and store it within an Azure Storage Table. This should give reasonably accurate tag and category clouds, unless you are publish a very large number of posts on a particular day compared to the total number of posts so far (or if you create new tags or categories during the day).  But the next day the clouds will be up to date again.

Your website can then retrieve the cached information from the Azure table for each view where you want to display the tag and/or category clouds.

For example within an MVC controller where you want the Home/Index and Home/ShowClouds view to show the tag and category clouds (but you could instead put the clouds in a sidebar):

Example MVC Controller: HomeController.cs

using ButterCMS; // added for ButterCMSClient
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration; // added for IConfiguration
using Microsoft.WindowsAzure.Storage; // added for CloudStorageAccount
using Microsoft.WindowsAzure.Storage.Table; // added for Table
using Project.Models;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading.Tasks;

namespace Project.Controllers
{
    public class HomeController : Controller
    {
        private ButterCMSClient client;
        private readonly string _connectionString;

        public HomeController(IConfiguration configuration)
        {
            var apiToken = configuration["ButterCMSApiKey"];
            client = new ButterCMSClient(apiToken);
            _connectionString = configuration["NameForYourAzureStorageAccountSetting"];
        }

        public async Task<ActionResult> Index()
        {
            return View();
        }

        public async Task<ActionResult> ShowClouds()
        {
            await GetTagAndCategoryCloudRecords();
            return View();
        }

        private async Task GetTagAndCategoryCloudRecords()
        {
            var storageAccount = CloudStorageAccount.Parse(_connectionString);
            var tableClient = storageAccount.CreateCloudTableClient();
            var tableName = "NameOfYourWordCloudsTable";
            var table = tableClient.GetTableReference(tableName);

            var tagCloudRecords = await ObtainListOfRecordsOfGivenType(table, "Tag");
            var categoryCloudRecords = await ObtainListOfRecordsOfGivenType(table, "Category");
            ViewBag.tagCloudRecords = tagCloudRecords;
            ViewBag.categoryCloudRecords = categoryCloudRecords;
            ViewBag.ShowSideBar = true;
        }

        private static async Task<List<WordCloudRecord>> ObtainListOfRecordsOfGivenType(CloudTable table, string type)
        {
            List<WordCloudRecord> list = new List<WordCloudRecord>();
            TableQuery<WordCloudRecord> query = new TableQuery<WordCloudRecord>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, type));
            TableContinuationToken token = null;
            do
            {
                TableQuerySegment<WordCloudRecord> resultSegment = await table.ExecuteQuerySegmentedAsync(query, token);
                token = resultSegment.ContinuationToken;

                foreach (WordCloudRecord entity in resultSegment.Results)
                    list.Add(entity);
            } while (token != null);
            return list;
        }

        // other views and methods ...
    }
}

Example View Page: ShowClouds.cshtml

@using Project.Models;

@{
    ViewData["Title"] = "Show Tag and Category Clouds";
}
<h1>@ViewData["Title"]</h1>

@{
    double minFontSize = 8;
    double maxFontSize = 22;
    List<WordCloudRecord> tagCloudRecords = new List<WordCloudRecord>();
    double largestFrequency = 1;
    double lowestFrequency = 1;
    if (ViewBag.tagCloudRecords != null)
    {
        tagCloudRecords = ViewBag.tagCloudRecords;
        largestFrequency = (from i in tagCloudRecords
                            select i.frequency).Max();
        lowestFrequency = (from i in tagCloudRecords
                           select i.frequency).Min();
    }
    List<WordCloudRecord> categoryCloudRecords = new List<WordCloudRecord>();
    double largestFrequency2 = 1;
    double lowestFrequency2 = 1;
    if (ViewBag.categoryCloudRecords != null)
    {
        categoryCloudRecords = ViewBag.categoryCloudRecords;
        largestFrequency2 = (from i in categoryCloudRecords
                             select i.frequency).Max();
        lowestFrequency2 = (from i in categoryCloudRecords
                            select i.frequency).Min();
    }
}

<div class="container mt-3">
    <h2>
        Category cloud
    </h2>
    <div class="media border p-3">
        <div class="media-body">
            @foreach (var item in categoryCloudRecords)
            {
                var fontSize = minFontSize;
                if (largestFrequency2 != lowestFrequency2)
                {
                    fontSize = minFontSize + (item.frequency - lowestFrequency2) / (largestFrequency2 - lowestFrequency2) * (maxFontSize - minFontSize);
                }
                string fontSizeAsString = fontSize + "pt;";
                <a href="/blog/category/@Uri.EscapeDataString(item.slug)" style="font-size:@fontSizeAsString">@item.RowKey</a>
            }
        </div>
    </div>
</div>

<div class="container mt-3">
    <h2>
        Tag cloud
    </h2>
    <div class="media border p-3">
        <div class="media-body">
            @foreach (var item in tagCloudRecords)
            {
                var fontSize = minFontSize;
                if (largestFrequency != lowestFrequency)
                {
                    fontSize = minFontSize + (item.frequency - lowestFrequency) / (largestFrequency - lowestFrequency) * (maxFontSize - minFontSize);
                }
                string fontSizeAsString = fontSize + "pt;";
                <a href="/blog/tag/@Uri.EscapeDataString(item.slug)" style="font-size:@fontSizeAsString">@item.RowKey</a>
            }
        </div>
    </div>
</div>

The results

And this is what the clouds look like in a sidebar:

From that page:

NET Framework 4.8 builds on previous versions of the .NET Framework 4.x by adding many new fixes and several new features while remaining a very stable product.

I look forward to using this soon.

For many years, I and colleagues have been aware that the treatment of maths in .NET (and even before that in C++) can lead to code failing silently.

That is: the code stops working normally, but instead of stopping and letting the user know that there is a problem, it continues running, but with incorrect (or at least unexpected) output.

This happens with several things including divide by zero (where the code returns infinity and continues running) and log of zero or a negative number (where the code returns NaN, not a number, and continues running).

The work around has been to be aware of these problems and use custom functions to detect the problem and report it as soon as it occurs.

If you (or your developers) don't do that, then your code could well be failing silently, which is bad because your output may be incorrect, and:

1. You or your customers may not notice it, and
2. Even if you notice it, you will have to step through your code to find the (possibly many) places where the problem occurs.

I had noticed that .NET Core 2.2 (and 2.1, and possibly earlier versions too) had fixed a problem with .NET 4.72 and earlier's Dictionary class, and had hoped that along with that fix, .NET Core would have also fixed the above maths problems. 

But on investigation, this does not seem to be the case.

I have published a short code repository on GitHub to show this - see here.

A problem with the Dictionary in .NET 4.72 and earlier

(For non technical readers, a Dictionary is a collection of key, value pairs, where the keys have to be unique.)

The Dictionary problem in .NET 4.72 and earlier is that if you ask for a value in the Dictionary corresponding to a key that doesn't exist (e.g. 2), then instead of telling you the useful information (e.g. "You asked for a value for key 2, which doesn't exist in the dictionary"), .NET 4.72 and earlier misses out the important information. It just says "The given key was not present in the dictionary".

See the repository for the full code, but a screenshot of the relevant code for .NET Core 2.2 Console App project is:

Other developers: I'd be interested to hear of your reactions to this (maybe I'm missing something and it isn't a problem, but at the moment I don't think so), and other ways of addressing the issues.

One example of how #AI can make it easier for your staff, customers or suppliers to interact with your software tools is to add a combined”Next Step / Tell me what you want to do” facility.

This uses natural language processing (NLP) combined with knowledge of who the user is (and what their role is, e.g. whether they are a member of staff, a customer, or a supplier, or a user with admin rights for example) and the context (which page or part of the app they are on, and what data they have stored in the system), to add two powerful new ways for the user to interact (with minimal training) with the app:

What’s my next step?

On any page, simply clicking the Go button asks the system “What’s my next step?”. The system then look intelligently at the user’s identity, role, data and location within the app and makes one or more suggestions as to what the user could usefully do next to make the most of the app.

For a couple of examples, please see here.

I am delighted to have now completed the Microsoft Professional Program for Data Science. It has been 10 online courses (taking a total of 322 hours) over just more than 11 months and my average (mean) mark over the 10 courses was 96.6%. The final course was a capstone project which involved analysing data from the 2015 earthquake in Nepal, building a model to help predict the degree of damage to buildings (amongst other things to help emergency response teams prioritise rescue efforts) and producing a report on this. This was an extremely practical way to complete the course.

I have created a series of slides (collected together in a Microsoft Sway online document) showing the main stages of my journey.  This should give you a pretty good flavour of the sort of work required on the course (and I hope will help people contemplating doing this or a similar course - I wish you the very best in your own data science journey!).

You can see the slides at https://sway.com/lsUjwGITuGFpsHIM?ref=Link

I have completed Microsoft's Professional Programs for both Data Science and Big Data, and I expect to complete their Professional Program for AI (Artificial Intelligence) in July 2019.

The particular problem that I have been using Azure Functions to address is an #actuarial #BigData #pensions one:

how to perform a financial valuation of large numbers (100,000 or even several million) of UK defined benefit pension plan members via cashflow projections as quickly as possible, but without incurring unnecessary cost .

When done accurately and before any clever processing, the number of cashflows produced can easily run into the millions, and for a large pension plan, into the billions. (When I have time, I plan to explain why, and talk about some of the clever processing that can be done).

This situation is one that obviously calls for parallel or distributed processing, with the work shared out between many machines.

The traditional (non serverless) way is to simply spin up as many machines as you like (e.g. I have done this with more than 150 cores), split the work amongst them and keep reallocating any remaining work amongst the machines until all the work has been done.  That can be done using Microsoft technology using #AzureBatch.  But it is also the most expensive way to do things, because all the machines are dedicated to your work until it is all completed. So naturally you pay for the number of cores times the time taken.

So, given that it takes some time for the machines to be ready for work, this is the pattern to be expected Azure Batch (caveat: I am extrapolating from my experience over larger workloads taking closer to 2 hours than here about 7 to 8 minutes, so have made a note to test Azure Batch over this smaller period):

It takes a couple of minutes for the instances to all become ready for use (but since they all started being warmed up from cold at the same time, there is little variation in the time to warm for them), then the machines are all fully occupied as the work gets allocated evenly to them, then (probably because of a little random variation in the speed each machine takes - perhaps dependent on relative health, e.g. out of 150 some may perform slightly worse than others) the bulk of the work gets done with a few stragglers taking slightly longer.

The serverless way is to use something like Azure functions, in which case one way of doing things (the way that I have used) is to split the work into small chunks and send lots of messages to an Azure queue, each one containing the instructions for a particular chunk of work.  

In theory what should then happen is that Microsoft Azure will realise that the queue is getting lots of messages, and start spinning up an increasing number of machines ("instances") to do the work, and keep doing this until all the messages in the queue have been dealt with.  

Of course, there is no guarantee that Azure will keep increasing the number of instances until the work is done. Instead, as I understand it, Azure uses an algorithm to determine how many instances are needed, and after increasing the instances for a while, may actually start decreasing them if the algorithm deems that the queue is going down satisfactorily enough.  (I have not been able to find out how the algorithm works, but it is said to take into account not only the rate of increase/or decrease of messages in the queue, but also the age of the oldest message in it.  This is further complicated by the fact that some messages - for reasons that are hard to find out - seem to fail to be processed at first, and are then hidden from the queue for several minutes, before becoming visible again in which case they might be processed.  As I will demonstrate, while this only seems to affect a small percentage of messages, it can delay the time until all the work is complete quite significantly.

I  have recently carried out some 91 different timed runs using Azure Functions under a variety of circumstances, e.g. in some I used Azure functions version 1, in some version 2. I also varied the number of queues (sometimes allocating all the work to 1 queue, sometimes to 2 queues, so that 2 different functions are performing the work, and in some to 5 queues.  In some of the runs with 2 or 5 queues I applied a form of load balancing (by moving messages from the slowest queues to the fastest ones), in others I did not.

I also repeated the same run over and over again, in order to test how repeatable runs using Azure functions are.  

I found that the answer (both in terms of the number of instances doing the work, and in terms of the total time taken to complete the work) sometimes varies quite a lot.

In all my runs, the task was the same: for each of 1000 queue messages, the Azure function processing it had to wait for 65 seconds and then it could return and report successful completion.

Here is an example of a situation when there was a reasonable degree of repeatability in terms of total time taken (although not in terms of how many instances performed the work): my runs 48 to 57 were a deliberate repeat of run 45, in which the work was allocated to 5 different queues, so equally to 5 different Azure version 2 functions, each with 200 messages to process.

Out of 11 runs (including run 45), the total time taken was pretty close across 10 of the runs, apart from one (run 53), where the work took considerably longer to complete:

NB the technologies I used in these runs  were Azure Queues, Azure Functions, Power BI, SQLServer (on completion of processing a message, each function logged its start time and end time and the identifier of the message to a separate queue, and a different type of azure function then saved that to a SQL Azure database, which was then processed by a web app to produce the type of chart above.  In addition, I used Power BI to check the approximate number of messages in each queue to monitor how quickly each queue was being dealt with.

More on this (and what exactly was happening in run 53?) in Part 2, which I hope to write in a few weeks time.

Microsoft's new Blazor technology is extremely interesting, because it offers the possibility of writing web apps using little or no Javascript.  

Until now, most websites typically require code to be written in at least two languages: Javascript for the client (browser) side, and C# (my preference, but others may prefer php, python, C++, VB.net, or other languages) for the server side.

That not only increases the learning curve, but also adds extra work when communicating between the server and the browser (e.g. dates have very different formats in Javascript and C#).  

So Blazor which is still only in Preview mode (like ASP.NET Core 3.0 and Visual Studio  2019 v 16.1 Preview 2 which are needed for it) looks as if it might significantly reduce the learning curve and pain of developing complex web apps.

The basic Blazor site available from Visual Studio 2019 v 16.1 Preview 2 comes with 2 small examples of interactivity that would normally be done using some javascript: updating a counter via clicking a button, and displaying data (retrieved from the server) in a table. 

Using Blazor for a more complex example: a blog with commenting allowed

As an experiment, I decided to try using Blazor for some something closer to a real world application and in the process to try out Telerik's new user interface components for Blazor (also in Preview, which of course makes sense because Blazor may change quite a bit before it is released). 

I wanted to try setting up a Blazor website that could be used not only for standard pages (of the sort in Microsoft's Blazor template), but that could also be used for blogging. For this purpose, I decided to try using a headless blog (i.e. one without a user interface), specifically ButterCMS.

I also wanted any visitors to be able to interact with the blog by leaving comments, so I decided to try using Disqus (one of the tools that ButterCMS recommend , since ButterCMS doesn't provide a commenting facility itself).

After a few gotchas (not surprising since all the products are in preview), I got things to work reasonably well.

For example, here is the Telerik TabStrip for Blazer in operation after clicking the dropdown list to tell it to put the tabs on the left:

and here is the Blazor template project's Fetch Data page amended to use the Telerik Grid for Blazor (I increased the number of records from 5 to 8 to showcase the grid's paging ability, and note that the Fahrenheit temperature column should be removed as the template project is not setting it  - it is zero in all cases):

Integrating with ButterCMS

ButterCMS provide helpful examples of how to call their  <abbr title="Application Programming Interface">API</abbr> in an ASP.Net MVC web app, and it only needed a few alterations to get this to work with Blazer's Razor pages.

Integrating with Disqus was a bit more difficult, because for the first time I had to resort to using Javascript, because that is how Disqus integrates with websites at the moment. (Maybe they'll write an API that can be called directly from Blazer C# code in future).  And that required using Interop between Blazer and Javascript.  But after some experiments and some very quick and useful help from Microsoft's Luke Latham (many thanks, Luke!) I got that working too:

A few problems (not surprisingly since all the technology is still in preview):

I did notice that if anything went wrong (e.g. i'd put a typo in the javascript - something that is much easier to do than in C# where Visual Studio is much more likely to pick it up before run time), then the Blazer navigation seemed to stop working. 

More significant is that so far I have been unable to get Bootstrap 4 styling to work properly in the Blazer project.  At first I thought this might be because I had had to change the css and js files from the Blazer default project ones to the ones needed by the Telerik UI components for Blazer.  But even after reverting to the default Blazer project ones, some basic things like getting navigation buttons/links to line up properly:

Some of the gotchas I encountered

When publishing to Azure, I needed to add the ASP NET Core 3.0 runtime extension to the app service to avoid an error (see my comment here on docs.microsoft.com). 

With regard to getting the javascript interop to work with Disqus, please the conversation here with Microsoft's very helpful Luke Latham.

Conclusion

I really like Blazor so far, and was very pleasantly surprised that it is already possible to produce quite a complex site (particularly the integration with ButterCMS and Disqus) already, while it is still in Preview.  I need to look into why the Bootstrap 4 formatting isn't working properly though.  When I have time (which I hope will be in a few weeks) I would definitely like to return to this!