Mozcode Blog - Software development, architecture and more

Building Transport‑Agnostic AI Agents with Semantic Kernel, A2A Protocol and Agent Discovery Service

2025-07-30T02:00:00-07:00

1. What is the Agent‑to‑Agent (A2A) protocol?

Originating from Google’s open specification, A2A defines a lean JSON‑RPC envelope that lets autonomous agents share intents, stream partial results and report progress in real time. By speaking A2A, you can mix and match agents written in any language or framework without inventing yet another bespoke API.

2. Why pair A2A with Microsoft Semantic Kernel?

Semantic Kernel (SK) gives C# developers a first‑class toolkit for wrapping LLM prompts as skills, composing them into pipelines and injecting traditional .NET code where it still shines. When you bolt A2A on top, each SK instance becomes a portable micro‑agent whose skills are instantly discoverable and callable by its peers.

3. Why discovery matters

The most important piece of the puzzle is not messaging or prompts — it's agent discovery.

The discovery service in this repository acts like a temporary memory of the agent ecosystem. Agents register their capabilities (like reverse or upper) with it, including the transport they support and how to reach them. Other agents then poll the /resolve endpoint until the desired skill becomes available, and only then initiate contact.

This approach brings three major benefits:

Loose coupling – Agents don’t need to know about each other ahead of time.
Late binding – You can add, replace, or scale agents without changing consumers.
Transport independence – The discovery layer is completely unaware of how agents talk.

Think of it as DNS for autonomous bots: discover first, communicate second.

4. What the repository contains

Agent2AgentProtocol.Discovery.Service – a minimal ASP.NET Core API that registers capabilities and resolves them to endpoints.
Semantic.Kernel.Agent2AgentProtocol.Example.Agent2 – the responder. It advertises two text‑processing skills: reverse and upper.
Semantic.Kernel.Agent2AgentProtocol.Example.Agent1 – the initiator. It hunts for an agent that can handle reverse then streams the outcome back to the console.
Semantic.Kernel.Agent2AgentProtocol.Example.Core – shared plumbing: the A2A message models, a pluggable IMessagingTransport, and helper extensions for SK.

5. Cloning and building


        git clone https://github.com/jordiag/semantic-kernel-agent-a2a-protocol-example.git
        cd semantic-kernel-agent-a2a-protocol-example
        dotnet build

6. Firing up the demo


        # 1️⃣ Start the discovery service
        dotnet run --project Agent2AgentProtocol.Discovery.Service

        # 2️⃣ Launch the responder (Agent 2)
        dotnet run --project Semantic.Kernel.Agent2AgentProtocol.Example.Agent2

        # 3️⃣ Launch the initiator (Agent 1)
        dotnet run --project Semantic.Kernel.Agent2AgentProtocol.Example.Agent1

Agent 2 registers its skills with the discovery service, including the chosen transport and address. Agent 1 polls /resolve until it finds a match, then dispatches a JSON‑RPC request such as reverse:Hello A2A. As the responder streams each chunk, the console shows something like: A|2|A | olleH.

7. Switching transports – zero code changes

The demo defaults to a local NamedPipeTransport for friction‑free testing. To see the exact same conversation traverse the cloud:

Provision an Azure Service Bus namespace and queue.
Set an environment variable called SERVICEBUS_CONNECTIONSTRING.
Flip UseAzureServiceBus = true in both agents’ appsettings.json.

Because both transports implement the same IMessagingTransport interface, no other code changes are required. Feel free to add WebSockets, gRPC or MQTT – the agents will not notice.

8. Anatomy of a capability card

Agent 2 serialises a concise JSON card that lists each skill, its description and the transport payload needed to reach it. Here’s a trimmed example:


        {
        "capabilities": [
        {
        "name": "reverse",
        "description": "Reverses input text",
        "transport": {
        "type": "NamedPipe",
        "address": "sk-agent2"
        }
        }
        ]
        }

9. Extending the sample

Add skills: Drop a new Semantic Kernel function into TextProcessingFunctions.cs, register it, and voilà – it appears in the capability card.
Experiment with transports: Plug in RabbitMQ or Redis streams by implementing IMessagingTransport.
Scale out discovery: Swap the in‑memory dictionary for Consul, etcd or an Azure App Configuration store.
Track long‑running tasks: Use the TaskManager helper in a2a-dotnet to stream incremental progress.

10. Key takeaways

By separating discovery, transport and skill execution, this tiny solution proves that AI agents can stay loosely coupled yet perfectly interoperable. Clone it, tweak it and watch your own agents strike up a conversation—without caring where or how their messages travel.

Connecting Agentic AI to Your Azure DevOps: Introducing the .NET Azure DevOps MCP Server

2025-06-30T02:00:00-07:00

1. Why wire AI agents straight into Azure DevOps?

Manual DevOps and CI/CD hand-offs slow down modern teams. By exposing Azure DevOps through the open Model Context Protocol (MCP), your LLM agents can raise work items, queue builds or approve releases on your behalf—no brittle REST glue or browser automation required.

2. What the server gives you today

The repository contains thin .NET libraries for every Azure DevOps “hub”, all surfaced as discoverable MCP tools:

Boards – create Epics, User Stories and Tasks
Repos – open PRs, add reviewers, merge or abandon
Pipelines – queue, cancel, retry and fetch logs
Artifacts – publish or yank packages
Test Plans – manage plans, suites and cases
Wiki – create pages or entire wikis

Everything runs on .NET 9 and talks to Azure DevOps via the official SDK or REST API, keeping the server implementation replaceable.

3. Getting the bits


        # clone and build
        git clone https://github.com/Jordiag/azure-devops-mcp-server.git
        cd azure-devops-mcp-server
        dotnet build
        # (optional) run the test suites
        dotnet test

4. Spinning up the server locally

The MCP server is a console app under src/Dotnet.AzureDevOps.Mcp.Server. Supply a personal-access token (PAT) and the URL of your Azure DevOps organisation as environment variables:


        export ADO_ORG_URL="https://dev.azure.com/contoso"
        export ADO_PAT=""
        dotnet run --project src/Dotnet.AzureDevOps.Mcp.Server

By default the server listens on http://localhost:5010 and serves an SSE endpoint that agents can subscribe to.

5. Talking to it from Semantic Kernel

Because each MCP tool is announced through function calling metadata, you can let Semantic Kernel auto-invoke them. The snippet below initialises an agent that calls the built-in echo test tool.


        // Program.cs
        using Microsoft.Extensions.DependencyInjection;
        using Microsoft.SemanticKernel;
        using Microsoft.SemanticKernel.Agents;
        using Microsoft.SemanticKernel.Connectors.OpenAI;
        using ModelContextProtocol.SemanticKernel.Extensions;

        var serverUrl = Environment.GetEnvironmentVariable("MCP_SERVER_URL")!;
        var openAiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
        var model = Environment.GetEnvironmentVariable("OPENAI_MODEL")!;

        IKernelBuilder builder = Kernel.CreateBuilder();
        builder.Services.AddOpenAIChatCompletion("openai", model, openAiKey);
        var kernel = builder.Build();

        // register every MCP tool exposed by the server
        await kernel.Plugins.AddMcpFunctionsFromSseServerAsync("ado-mcp", serverUrl);

        var settings = new OpenAIPromptExecutionSettings
        {
        ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
        };

        var agent = new ChatCompletionAgent
        {
        Name = "DevOpsBot",
        Kernel = kernel,
        Instructions = "Use available tools to answer the user.",
        Arguments = new KernelArguments(settings)
        };

        await foreach (var update in agent.InvokeAsync(
        "Queue the build definition #123 in project Contoso and return the run URL."))
        {
        Console.WriteLine(update.Message);
        }

The LLM interprets the plain-English instruction, discovers the queuePipeline tool and fires the build without you writing explicit API calls. Magic.

6. Under the bonnet: tests and quality gates

The repo ships integration and full agent end-to-end tests that hit a real Azure DevOps org, plus SonarCloud analysis for maintainability, security and coverage badges.

7. Final thoughts

If you’re already experimenting with agentic workflows, the Azure DevOps MCP Server hands your bots a first-class CI/CD toolbox while keeping credentials on your machine. Give it a try, star the repo and open an issue with your feedback—new ideas land here weekly.

How Microsoft.Extensions.AI Complements Semantic Kernel: A Deeper Dive

2025-05-01T02:00:00-07:00

1. Introduction

Semantic Kernel (SK) gives .NET developers an orchestration layer for building intelligent agents—prompt templates, plugins, planning, memory and more. Microsoft.Extensions.AI (ME.AI) sits one level lower, supplying lightweight contracts and a middleware pipeline for talking to any large-language-model back-end. By standing on ME.AI, SK inherits the same design principles that turned ASP.NET Core, Entity Framework Core and the wider Microsoft.Extensions stack into reliable foundations: dependency injection as a first-class citizen, composable middleware and testability out of the box.

2. Layered mental model

If you imagine building a web API, SK is equivalent to the MVC layer (controllers, views, filters) whereas ME.AI resembles the underlying HTTP abstractions such as HttpClientFactory. The following ASCII sketch places each piece in context.


        ┌───────────────────────────────────────────┐
        │          Your Chatbot / Copilot           │
        └──────────────▲────────────────────────────┘
        │ Plans, plugins, vector-memory
        ┌──────────────┴────────────────────────────┐
        │            Semantic Kernel (SK)           │
        │  (prompts, planning, agents, memories)    │
        └──────────────▲────────────────────────────┘
        │ IChatClient, IEmbeddingGenerator
        ┌──────────────┴────────────────────────────┐
        │         Microsoft.Extensions.AI           │
        │ (abstractions, DI, middleware, logging)   │
        └──────────────▲────────────────────────────┘
        │ HTTP / gRPC / WebSockets
        ┌──────────────┴────────────────────────────┐
        │        OpenAI, Azure AI, Ollama…          │
        └───────────────────────────────────────────┘

3. Dependency injection fundamentals

All ME.AI clients register with the same IServiceCollection you already use in ASP.NET Core. This means every SK feature that depends on a model—chat, embeddings or function calling—can resolve its collaborators through standard DI without any bespoke static factories.


        // Program.cs or Startup
        var kernel = Kernel.CreateBuilder()
        .AddOpenAIChatClient("gpt-4o", Environment.GetEnvironmentVariable("OPENAI_KEY"))
        .AddAzureOpenAIEmbeddingGenerator("text-embed", "https://my-ai.azure.com/", azureKey)
        .Build();

        // Anywhere later in the app
        var chat = kernel.Services.GetRequiredService();
        Console.WriteLine(await chat.GetResponseAsync([new(ChatRole.User, "Hello, world!")]));

Notice there is no SK-specific chat interface—SK leans directly on ME.AI.

4. Middleware and cross-cutting concerns

ME.AI borrows ASP.NET Core’s .UseXxx() convention, so you can slot in functionality such as caching, rate-limiting or telemetry with a single line and without touching business logic.


        var client = new ChatClientBuilder(new OpenAIChatClient("gpt-4o", key))
        .UseDistributedCache(redisCache)          // transparent response caching
        .UseRetryPolicy(RetryPolicy.Exponential())// resilience
        .UseOpenTelemetry(sourceName: "ai-demo")  // traces, metrics, logs
        .Build();

Because SK consumes that IChatClient, every plugin and planner automatically benefits from the middleware you configure.

5. Observability: traces, metrics and logs

OpenTelemetry support in ME.AI emits structured spans for each model call. You get duration, tokens in/out, temperature and any tool invocations—all correlated with the rest of your application traces. Hook it up to Application Insights, Jaeger or Zipkin and you can follow a request from an HTTP endpoint, through SK planning, down to the individual LLM round trips.

6. Caching and performance

LLM calls are often the slowest and most expensive hop. ME.AI’s built-in distributed-cache middleware lets you memoise prompts on Redis, SQL or even in-memory caches. When SK asks the same question twice, the second hit can be served in microseconds instead of seconds.

7. Multiple providers—a practical example

It is common to combine a hosted model such as OpenAI for production with a local model like Ollama for offline development. Service IDs make that trivial.


        services.AddOpenAIChatClient("gpt-4o", openAiKey, serviceId: "Prod");
        services.AddOllamaChatClient("llama3", new Uri("http://localhost:11434"), serviceId: "Dev");

        var kernel = services.BuildServiceProvider().GetRequiredService();

        // Use OpenAI in production paths
        Console.WriteLine(await kernel.InvokePromptAsync("Say hi", new(new PromptExecutionSettings { ServiceId = "Prod" })));

        // Switch to local model for quick dev inner loop
        Console.WriteLine(await kernel.InvokePromptAsync("Say hi", new(new PromptExecutionSettings { ServiceId = "Dev" })));

8. Testing strategies

Because clients are plain interfaces, you can inject a stub or mock when running unit tests. This keeps test runs fast, deterministic and offline.


        // xUnit test
        services.AddSingleton(new FakeChatClient("🚀")); // returns a canned response
        var kernel = services.BuildServiceProvider().GetRequiredService();

        string result = await kernel.InvokePromptAsync("ignored");
        Assert.Equal("🚀", result);

9. Security and governance

Enterprises often require content filtering, data-loss-prevention checks and policy enforcement before anything leaves the network. Implement that logic once in an IChatClient decorator or a middleware component. All downstream SK features inherit the same guard rails.

10. Migration at a glance

Swap any custom HttpClient or model SDK calls for ME.AI client registrations.
Remove obsolete SK-specific interfaces. Let SK resolve IChatClient and IEmbeddingGenerator through DI.
Add optional middleware: caching, OpenTelemetry and retry policies.
Write unit tests against the new abstractions using stubs or mocks.

11. Final thoughts

Microsoft.Extensions.AI acts as the standard plumbing layer for generative AI in .NET. Semantic Kernel builds orchestration, planning and agent logic on top of that foundation. By embracing both, you gain clean dependency injection, composable middleware, first-class observability and lighter tests—while keeping all the high-level features that made SK popular in the first place.

Semantic Kernel and the Model Context Protocol: Building the Future of AI Agents

2025-04-27T02:00:00-07:00

Introduction

In the rapidly evolving landscape of artificial intelligence, the integration of Microsoft's Semantic Kernel (SK) with the Model Context Protocol (MCP) represents a significant advancement in building intelligent, context-aware AI agents. This synergy enables developers to create agents that can seamlessly interact with diverse data sources and tools, enhancing their capabilities and adaptability.

Understanding Semantic Kernel

Semantic Kernel is an open-source SDK developed by Microsoft that facilitates the integration of AI services into applications. It provides a modular and extensible framework for building AI agents capable of performing complex tasks by combining natural language understanding with programmable functions. Key features include:

Plugins: Modular components that encapsulate specific functionalities, enabling agents to perform a wide range of tasks.
Memory: Allows agents to retain context over time, improving their ability to handle multi-turn conversations and complex workflows.
Planning: Enables agents to decompose tasks into sub-tasks and execute them in a structured manner.
Function Calling: Allows agents to invoke external functions or APIs based on user input or internal logic.

Introducing the Model Context Protocol (MCP)

MCP is an open standard introduced by Anthropic in late 2024 to standardize how AI models interact with external tools and data sources. By leveraging JSON-RPC 2.0, MCP provides a universal interface for AI assistants to read files, execute functions, and handle contextual prompts. Its adoption by major AI providers, including OpenAI and Google DeepMind, underscores its significance in the AI ecosystem.

The Synergy of Semantic Kernel and MCP

Integrating Semantic Kernel with MCP allows developers to build AI agents that can interact with a wide range of tools and data sources through a standardized protocol. This integration offers several benefits:

Interoperability: MCP provides a consistent interface for AI agents to interact with various tools, reducing the complexity of integrations.
Scalability: Developers can build agents that scale across different platforms and services without rewriting integration logic.
Flexibility: The combination allows for dynamic function calling and context-aware interactions, enhancing the agent's capabilities.
Security: MCP's design includes considerations for secure data exchange, ensuring that integrations adhere to best practices.

Real-World Applications

By integrating SK with MCP, organizations can develop AI agents capable of:

Automating customer support by accessing CRM systems and knowledge bases.
Streamlining internal processes through interaction with enterprise resource planning (ERP) tools.
Enhancing data analysis by connecting to various databases and analytics platforms.
Facilitating natural language interactions with complex systems, improving user experience.

Getting Started with Integration

To begin integrating Semantic Kernel with MCP:

Set up your development environment with the latest versions of SK and MCP SDKs.
Define the functions and data sources your AI agent needs to access.
Implement MCP servers or clients as needed to facilitate communication.
Utilize SK's plugin architecture to register and manage these functions within your agent.

Security Considerations

While the integration of SK and MCP offers numerous benefits, it's essential to address potential security concerns. Developers should implement robust authentication and authorization mechanisms, validate inputs, and monitor interactions to prevent unauthorized access and ensure data integrity.

Conclusion

The integration of Semantic Kernel with the Model Context Protocol marks a significant step forward in the development of intelligent, context-aware AI agents. By leveraging the strengths of both technologies, developers can create solutions that are not only powerful but also adaptable to the ever-changing landscape of enterprise needs.

For more information, refer to the official documentation:

Introduction to Semantic Kernel: AI-powered agents for your apps

2025-04-20T02:00:00-07:00

What is Semantic Kernel?

Semantic Kernel is a lightweight and modular open-source SDK from Microsoft that helps developers integrate large language models (LLMs) into their applications in a seamless and extensible way. By acting as a bridge between your application logic and the power of modern AI services like OpenAI and Azure OpenAI, it lets you build intelligent agents that understand context, plan actions, and execute real-world functions.

Why Use Semantic Kernel?

While many SDKs enable chat or text generation, Semantic Kernel goes further by offering a framework to define and organise your app's capabilities as plugins, which the AI can invoke. It’s designed with enterprise needs in mind—telemetry, logging, and modular design come built-in. It supports automatic function calling, multi-turn dialogue, and integration with tools like Microsoft 365 Copilot.

Core Components Explained

Kernel: The container managing services and plugins. It acts as the central orchestrator for your AI agent.
Connectors: Interfaces to AI services such as OpenAI, Azure OpenAI, or Hugging Face.
Plugins: Your app logic, APIs, or business functions exposed to the AI agent for action.
Prompt Templates: Predefined text templates used to generate consistent prompts for the AI models.
Native Functions: Application-level logic that can be invoked by the AI using function calling.

Function Calling and Plugins

One of the most impressive features is the automatic function calling. You describe your function with attributes, and when the AI recognises that a user’s intent maps to that function, it automatically invokes it with the appropriate parameters. This allows for intelligent action, not just conversation.

Getting Started in .NET

Here is a brief setup guide for C# developers:

dotnet new console -n SemanticKernelDemo
        cd SemanticKernelDemo
        dotnet add package Microsoft.SemanticKernel

Then define your kernel and chat service like so:

using Microsoft.SemanticKernel;
        var builder = Kernel.CreateBuilder();
        builder.AddAzureOpenAIChatCompletion("model", "endpoint", "apiKey");
        Kernel kernel = builder.Build();

Real-World Use Cases

Developers are using Semantic Kernel in exciting ways:

Automated email drafting with attachments and scheduling.
Customer service bots that not only respond but trigger actions (like order status retrieval).
Internal business process automation, e.g., expense report generation.
IoT control via AI—turning on lights, checking sensors through plugins.

Advanced Features

Planning: The SDK supports plan generation, where the AI outlines a multi-step execution path (e.g., "Get user data, then send a report"). This is especially powerful when combined with memory or context-aware functions.

Observability: You can monitor all interactions using built-in logging and telemetry. Hooks and filters allow developers to execute pre/post logic around prompts or function calls, enabling advanced auditing and compliance support.

Supported Languages and Extensibility

Semantic Kernel is available in C#, Python, and Java, and it includes a rich plugin architecture. You can build plugins from:

Native classes with annotated methods
Prompt templates defined in YAML or Prompty format
OpenAPI specifications for REST APIs

Community and Resources

Semantic Kernel is actively developed and has an open-source community. You can find learning resources including sample notebooks, official guides, and detailed examples on GitHub:

https://github.com/microsoft/semantic-kernel

Agentic AI with Semantic Kernel

Agentic AI refers to artificial intelligence systems that can act autonomously to achieve user goals. Instead of merely responding to prompts, these systems assess context, formulate a plan, and take initiative to complete tasks. Semantic Kernel is purpose-built for such scenarios.

By combining planning capabilities, function calling, plugin extensibility, and memory integration, Semantic Kernel enables developers to construct agents that go beyond chat. For example, an agent can interpret a request like “generate a report from yesterday’s sales and email it to my manager” by breaking it down into steps and executing them without further input.

These kinds of autonomous, reasoning-driven agents are the future of enterprise software, and Semantic Kernel provides all the foundational components to build them today.

Final Thoughts

Semantic Kernel isn’t just another AI SDK—it’s a strategic foundation for building intelligent, modular, and actionable agents. Whether you’re prototyping, building internal tools, or deploying full-scale AI solutions, it’s an essential addition to your AI toolbox.

Free Azure DevOps CICD with Smarterasp.net hosting, Part 2

2020-12-12T03:06:12-08:00

Description

On my previous post I started to explain how to create a full CI/CD DevOps automated deployment cycle for your Smarterasp.net hosting websites defining the build pipeline implementation.

After creating a Build pipeline, we are going to finish it creating a release pipeline.

Requirements

You will need to fulfil some requirements first:

Previous post regarding build pipeline already done.
An SmarterAsp web hosting.

Steps to create a release pipeline

Check if the .NET version of your website is supported in smarterasp https://www.smarterasp.net/support/kb/a1986/supported-versions-of-_net-core.aspx
End of support versions normally are disabled, check here which version is already deprecated:
https://dotnet.microsoft.com/platform/support/policy/dotnet-core

Login to your smarterAsp.net hosting account control panel, go to [websites] tab. If there is no website already created, create one clicking [+New website] button and follow the steps, it is pretty straightforward.

After that you can expand your website settings clicking on [***] button on the right, you will have to check two main settings: Asp .NET Version and VS Webdeploy:

Check that your website has assigned the right pool type (ASP.NET 4.x Integrated):

Check that your VS Webdeploy feature is enabled and copy: Service URL, Site/Application and Username in a temporary place like your notepad, you will use them later (password is the same you use to enter Smarterasp.net control panel):

In Azure Devops check that your build pipeline generates the scripts to deploy your website using Web Deploy. Go to [Pipelines]-> Click on your [build pipeline name]-> Click on the [latest build]-> you will see a table where there is a row named [Related: 1 published(...)], click on 1 published:

You should be able to see a folder with a [xxx.cmd] file, some [xxx.xml] files and a [xxx.zip] file similar to this screenshot:

After confirming that, you ensured you have the necessary files from your build pipeline, now let's start in Azure DevOps with the release pipeline work, open [Releases] tab in [Pipelines] section and click new release:

Select [Empty job] template:

If you don't plan to create a multi environment pipeline like [Development -> Staging -> Production]... just leave it with one basic environment: [Stage 1] and close this form:

Add the artifact generated by your build pipeline, click [+Add]:

Select the source (build pipeline) from this drop down and click [Add]:

Set [continuous deployment trigger] to [Enabled] so every time a build is finished will trigger an automatic deployment:

From the notepad details you saved before, create a new release variable for your password, click on [variables] and add: smarteraspPassword variable name, value, scope and save:

Now, let's add some deployment tasks to [Stage 1] environment, click on [1 job, 0 tasks]:

Firs add two [RegEx Find & Replace task], in the search box type: replace, a specific task box will be shown, Add it two times:

Add a PowerShell task too:

You should have 3 tasks added, let's configure the first one.
1) Add a meaningful name in Display name.
2) In Search Paths to Input Files field add: **\yoursite.deploy.cmd
where you have to replace the bold part with the real name of the cmd file generated
by your build release as you have seen before in the screenshot above.
3) In Find RegEx field add: _ArgComputerNameWithQuote\="%_ArgValue%"&goto
4) In Replace With field add: _ArgComputerNameWithQuote="https://yoursite:8172/
msdeploy.axd?site=yoursite-001-site4"&goto
where you have to replace the bold part with the real Service URL value you copied in
notepad from SmarterAsp.net Web Deploy details before.

NOTE: SmarterAsp.net creates a self-signed certificate for the WebDeploy port 8172.

Let's configure the second Replace task.
1) Add a meaningful name in Display name.
2) In Search Paths to Input Files field add: **\yoursite.SetParameters.xml
where you have to replace the bold part with the real name of the xml file in your build as
you've seen before in the screenshot above.
3) In Find RegEx field add: \<setParameter name\="IIS Web Application Name"
value\="Default Web Site" \/\>
4) In Replace With field add: <setParameter name="IIS Web Application Name"
value="yoursiteapp" />
where you have to replace the bold part with the real Site/Application value you copied in
notepad from SmarterAsp.net Web Deploy details before.

Let's configure the last step, the powershell script to deploy your website.

1) Add a meaningful name in Display name.
2) In Script field add: your-file-path/yoursite.deploy.cmd /Y /M:hardcoded /U:your-user-001 /P:$(smarteraspPassword) /G:False /A:Basic -allowUntrusted -enableRule:AppOffline

where you have to replace your-file-path with the real path of the xml file in your build as
you've seen before in the screenshot above. (In my case was:
_mozcode.com-ASP.NET-CI/drop/Miniblog.Core.deploy.cmd)

Replace your-user-001 with the real username as you've seen before in the
screenshot.

(Note: you are using the release password variable you declared before and getting it value
in this way: $(smarteraspPassword) ).

3) Click [Save]and then click [Create Release]

You should be able to see a successful web deploy release in your pipeline log:

If you see Error 500.34 in your browser after deploying your website and checking how it looks it happens because SmarterAsp.net is sharing your web application pool with another existing website and it won't work in this way. More info in these links:

https://www.smarterasp.net/support/kb/a1999/what-should-we-do-when-get-http-error-500_34-ancm-mixed-hosting-models-not-supported.aspx

https://weblog.west-wind.com/posts/2020/Jan/14/ASPNET-Core-IIS-InProcess-Hosting-Issues-in-NET-Core-31

To fix it, you have to add this line (AspNetCoreHostingModel one) in your .csproj file in all your smarterAsp.net websites deployed to that shared application pool and deploy them again:

And that's all folks!

Free Azure DevOps CICD with Smarterasp.net hosting, Part 1

2020-12-07T03:46:04-08:00

Description

For personal, experimental, prototyping or small side projects lots of ASP.NET developers choose a cheap web hosting supplier like Smarterasp.net. If you are one of these developers using SmarterAsp.net for your projects nothing is stopping you to have a full CI/CD DevOps automated deployment cycle for your websites.

Traditionally the protocol to upload content to this kind of hosting plans is FTP or WebDeploy. Microsoft is offering a great DevOps free tool: Azure Devops. Setting it up is quite easy, just follow the steps after clicking [Start for free] button.

Once you have your Azure Devops project set, you will be able to configure and execute build and release pipelines for free targeting your Smarterasp.net web hosting plan.

Requirements

You will need to fulfil some requirements first:

Azure DevOps already set with an empty project.
An SmarterAsp web hosting.
A .NET website to deploy.

Steps to create a build pipeline

Start going to your Azure DevOps portal, select your project and click in Pipelines -> Pipelines -> [Create Pipeline]:

In this case, to simplify, we won't use a YAML. We well use the classic editor which you can convert to YAML later:

Select a repo:

Select the ASP.NET template:

You can see a new build pipeline with some pipeline tasks set by default. First, set your agent to Windows latest version (in this moment 2019):

Now set your nuget excutable to the latest version available (in this moment 5.x):

All tasks should work OK now with the other default configuration values. After this, click "Save & queue". Congratulations! you finalised the CI part, now you have a successful build pipeline:

The next step is to create a release pipeline which will take the generated artifact and it will deploy it to your SmarterAsp.net hosting plan.

This is what is going to be described in the next article "part 2". Until then, enjoy!

C# .NET High performance Span demo

2020-11-20T03:05:41-08:00

Description

Span<T> is a new ref struct introduced in C# 7.2 specification. It is a stack-only type that allows memory operations without allocation so, used for instance in very large arrays, it can be a significant performance improvement.

It is only applicable if your code is based from .NET Core 2.1 and .NET Standard 2.1. There are tons of technical documentation available about Span<T>, this post is just going to be focused in a practical demo to compare the performance of the slice method. Span<T> can't work inside Anync methods but you can work around this issue easily creating a non Async local method.

Detailed information can be found in the official Microsoft link: https://docs.microsoft.com/en-us/dotnet/api/system.span-1?view=net-5.0

The source code of this demo is fully available to be downloaded and ready to be executed from my Github repo: https://github.com/Jordiag/high-performance-span-of-t-demo

Comparison code

Let's compare span<T> slice() method. We are going to calculate the sum af the values in an array skiping the first position. Easy right? We will compare 4 code versions doing the same operation in different ways.

(V1) The first version is using LINQ:

public partial class Calculation
{
	private readonly int[] intArray = new int[] { 1, 2, 3, 4, 5 };
	private static readonly Consumer Consumer = new Consumer();

	public int PartialArraySumV1()
	{
		intArray.Consume(Consumer);
		var arraySum = intArray.Skip(1).Take(4).Sum(x => x);

		return arraySum;
	}
}

(V2) After impementing it you think, "I know that sometimes LINQ is not super efficient, let's write it in a traditional loop":

public partial class Calculation
{
	public int PartialArraySumV2()
	{
		int result = 0;
		var arraySmall = intArray[1..intArray.Length];
		for (int x = 0; x < arraySmall.Length; x++)
		{
			result += arraySmall[x];
		}

		return result;
	}
}

(V3) Then, you realise that C# 7.2 intruduced a new feature: Span<T>, so why not to try it?, at the end you know that in production your engine is going to manipulate an array of 10 million positions so, the most performant, the better. Let's use slice() method instead:

{
    public int PartialArraySumV3()
    {
        Span<int> arraySpan = intArray;
        int result = 0;
        var arraySmall = arraySpan.Slice(1, intArray.Length - 1);
        for (int x = 0; x < arraySmall.Length; x++)
        {
            result += arraySmall[x];
        }

        return result;
    }
}

(V4) Gosh!!!, you are a real perfectionist, yesterday you read that C# 8.0 allows to use Span<T> slice() method with abbreviation which looks cool:

public partial class Calculation
{
    public int PartialArraySumV4()
    {
        var arraySpan = intArray.AsSpan();
        int result = 0;
        var arraySmall = arraySpan[1..intArray.Length];
        for (int x = 0; x < arraySmall.Length; x++)
        {
            result += arraySmall[x];
        }

        return result;
    }
}

How to compare performance?

Now the problem becomes, which one is more performant? Which one should you use when your project is already screaming for maximum performance?

Setting up DotNetBenchmark

Easy, let's use BenchmarkDotNet. We will add the BenchmarkDotNet nuget package in our benchmark console project:

Install-Package BenchmarkDotNet

We will add the benchmark execution line in program.cs main method:

namespace Engine.PerformanceBenchmarks
{
    internal class Program
    {
        static void Main(string[] args)
        {
            BenchmarkRunner.Run<Calculation>();
        }
    }
}

Setting up our calculation code for BenchmarkDotNet

We will decorate the Calculation class once with [MemoryDiagnoser] and all PartialArraySumVx methods with: [Benchmark] attribute:

[MemoryDiagnoser]
public partial class Calculation
{
    [Benchmark]
    public int PartialArraySumV1()
    {
       (***)
    }
}

public partial class Calculation
{
    [Benchmark]
    public int PartialArraySumV2()
    {
       (***)
    }
}

public partial class Calculation
{
    [Benchmark]
    public int PartialArraySumV3()
    {
       (***)
    }
}

public partial class Calculation
{
    [Benchmark]
    public int PartialArraySumV4()
    {
       (***)
    }
}

Executing the benchmark in Visual Studio

1) Set the console application benchmark project as startup project.

2) Set Visual Studio to target release build.

3) Run!

After waiting few minutes depending on your computer specifications, you will get these results in the console:

Benchmark conclusions

- Span<T> used in PartialArraySumV3 and PartialArraySumV4 is the most performant way using around 3 nanoseconds.

- No significant performance difference between slice() code declaration and slice abbreviation (V3 and V4).

- The traditional for loop wasn't that bad, but still 8.5 times slower than Span<T> implementations.

- LINQ was the worst performant option, 100 times slower than Span<T> and 8.5 times slower than a classic for loop.

Every coding situation is a different story but if you measure your code solution alternatives, you can see here that Span<T> could be a great performance asset.

Installing Hadoop HDFS data lake in Raspberry Pi 4 Ubuntu cluster

2020-07-11T03:41:15-07:00

Introduction

Weeks ago I decided to start creating an experimental home size "Big data" system based on Apache Spark. The first step for it is to create a distributed filesystem where Apache Spark will read and write eveything.

HDFS is the Hadoop distributed filesystem which provides features like: fault detection and recovery, huge datasets, hardware at data, etc... despite it is a Hadoop ecosystem piece, it works nice as the data distributed filesytem for Apache Spark.

The HDFS filesystem will be installed in 8 RPIs of my 10 Raspberry Pi 4 cluster. To have a minimal storage size and decent speed I bought 8 extra SSD + 8 USB3 adaptors.

Installing HDFS

First of all, we are going to install a HDFS filesystem in a single node to test everything works in a simple way. I already have Ubuntu 18.04 server for Raspberry Pi 4 installed but I guess these steps won't differ too much if you have Ubuntu 20.04 server installed.

In my case pi3cluster is the RPI master host name and pi4cluster, pi5cluster, pi6cluster, pi7cluster, pi8cluster, pi9cluster, pi10cluster are the node host names. All my raspberry pis have declared a /etc/hosts file with all the name/ip pairs, you must do it as well if it's not already done:

192.168.1.32        pi3cluster
192.168.1.33        pi4cluster
192.168.1.34        pi5cluster
192.168.1.35        pi6cluster
192.168.1.36        pi7cluster
192.168.1.37        pi8cluster
192.168.1.38        pi9cluster
192.168.1.39        pi10cluster

1. Lets prepare our new 8 SSDs. After plugging them all into the Raspberry PI 4 USB3 ports, they become available as the /dev/sda device. Let's partition and format all of them. Repeat these commands in all of your RPIs:

sudo fdisk /dev/sda

    #(select option n, new partition)
    #(select option p, primary type)
    #(select option w, write and exit)
    # like this:

2. format /dev/sda1 new partition and create a new linux directory to mount it there (do it in all of your RPIs)

sudo mkfs.ext4 /dev/sda1
sudo mkdir /mnt/hdfs

3. Edit your linux filesystem list and add the new one (do it in all of your RPIs)

sudo nano /etc/fstab

4. Once you have nano displaying it, add this line at the end of the file (do it in all of your RPIs):

/dev/sda1 /mnt/hdfs ext4 defaults 0 0

5.refresh your linux filesystem mounting points:

sudo mount -av

6. Install Java and wget packages, here: Hadoop Java Versions documentation you can find which version is recommended. (do it in all of your RPIs)

sudo apt install openjdk-8-jdk-headless wget

7. Add a new system user for your hadoop hdfs (do it in all of your RPIs)

sudo adduser hadoop

8. Download and copy the latest hadoop available package and install it only in your master RPI

cd ~
wget https://www-eu.apache.org/dist/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
tar xzf hadoop-3.2.1.tar.gz
rm hadoop-3.2.1.tar.gz
sudo mkdir /opt
sudo mv hadoop-3.1.2 /opt/hadoop
sudo chown hadoop:hadoop -R /opt/hadoop

9. Create hadoop hfds data folders (do it in all of your RPIs)

sudo mkdir -p /mnt/hdfs/datanode
sudo mkdir -p /mnt/hdfs/namenode
sudo chown hadoop:hadoop -R /mnt/hdfs

10. Login as hadoop user and create the ssh key (do it in all of your RPIs)

sudo -i
su hadoop
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

11. From your master node, logged in as hadoop user, authorise your master to all the other nodes adding your master ssh key as an authorised one, repeat this command for all your nodes replacing <ip> by your node ip

$ cat ~/.ssh/id_rsa.pub | ssh hadoop@<ip> 'cat >> .ssh/authorized_keys'

12. Add all environment variables needed by hadoop in this file /home/hadoop/.bashrc (do it in all of your RPIs)

export HADOOP_HOME=/opt/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

after adding those lines, reload your variables

source ~/.bashrc

13. Add Hadoop to your PATH, edit /home/hadoop/.profile. Add this line at the end of the file (do it in all of your RPIs)

PATH=/opt/hadoop/bin:/opt/hadoop/sbin:$PATH

14. Set JAVA_HOME in the hadoop environment config file

nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Add export java_home=/usr/lib/jvm/java-8-openjdk-arm64/bin/java like:

15. Let's start changing the hadoop cluster configuration files, you can find all texts to copy and paste here hdfs-config.txt. Add the configuration section to core-site hadoop file

nano $HADOOP_HOME/etc/hadoop/core-site.xml

16. Add the configuration section to hdfs-site hadoop file

nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

17. Add the configuration section to mapred-site haddop file

nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

18. Add the configuration section to yarn-site hadoop file

nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

19. After everything is configured and the SSD is partitioned and mounted, we can format the HDFS (only the master RPI)

/opt/hadoop/bin/hdfs namenode -format -force

20. Let's start HDFS filesystem and do some operations to check everything works fine

cd $HADOOP_HOME/sbin/
./start-dfs.sh
./start-yarn.sh
hadoop fs -mkdir /test_folder
hadoop fs -ls /

21. Check you can see all Hadoop info websites: (A) services web, (B) cluster info web, (C) hadoop node web, and (D) check the "test_folder" you created is there too

http://<your server ip>:9870/
http://<your server ip>:8042/
http://<your server ip>:9864/
http://<your server ip>:9870/explorer.html

(A)

(B)

(C)

(D)

21. If you want to restart the server later, use these commands

cd $HADOOP_HOME/sbin/
stop-all.sh
start-all.sh

From single node to HDFS cluster

After checking Hadoop HDFS working fine as a single node, It's time to extend it to all our capable nodes creating a HDFS cluster.

22. Let's start changing the hadoop cluster configuration files again, leave them as they are shown in the pictures. You can find all texts to copy and paste here hdfs-cluster-config.txt. Remember to change my RPI host names like pi3cluster to your specific names.

nano $HADOOP_HOME/etc/hadoop/core-site.xml

nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

23. In your cluster make sure all services are stopped, delete all files from before

cd $HADOOP_HOME/sbin/
stop-all.sh
$ clustercmd rm –rf /opt/hadoop_tmp/hdfs/datanode/*
$ clustercmd rm –rf /opt/hadoop_tmp/hdfs/namenode/*

24. Logged in as hadoop user, create master and workers files inside hadoop configuration.

nano /opt/hadoop/etc/hadoop/master
nano /opt/hadoop/etc/hadoop/workers

In master file, add only one line, your master RPI host name, in my case it looks like:

pi3cluster

In workers file, add as many lines as you need listing all the host names of your RPI nodes, in my case it looks like:

pi4cluster
pi5cluster
pi6cluster
pi7cluster
pi8cluster
pi9cluster
pi10cluster

25. In your master node, and ONLY there, format your name node. IMPORTANT: backup your hdfs files folder if you have important files there from before, they will be deleted.

hdfs namenode -format -force

26. From your single master node, tar your hadoop home to a shared network folder or a pen drive

sudo tar -czvf /mnt/shared-network-folder/hadoop.tar.gz /opt/hadoop

27. In all your new raspberry pi cluster nodes, extract the tar file in the same place and force the ownership to hadoop user

sudo tar -xzvf /mnt/shared-network-folder/hadoop.tar.gz -C /opt/
sudo chown hadoop:hadoop -R /opt/hadoop

28. From your master node, logged in as hadoop user start the HDFS system

start-dfs.sh && start-yarn.sh

29. You should be able to see your multinode hadoop HDFS distributed filesystem, browse this URL: http://<your-master-ip-or-node-name>:9870/dfshealth.html#tab-datanode

RPi 4 K8s multi-master cluster using Ubuntu and Kubespray

2020-01-26T08:48:00-08:00

Introduction

Recently, I've been thinking about improving my Kubernetes (K8s) Raspberry Pi cluster to something more resilient than just a one K8s master cluster which can fail easily and ruin all your work of months on it.

I installed multiple times on-premise single master K8s clusters based on Redhat ecosystem ( Centos/Fedora). The difficulty of doing this manually if it's your first time it's far from easy, and if you want to install something closer to an on premises HA K8s cluster manually it becomes a quite uncomfortable task.

So the challenge was to find a simple way to install a multi-master K8s cluster using my new 10 raspberry Pi 4 (4Gb Ram) project.

--post-breaker--

This article intends to share my experience doing this task as a quick and easy guide for anybody who wants the same results without loosing hours on it. The hardware environment will consist in a simple network like this one:

OS/Software stack

The keys to reach this goal was to use:

- Kubespray K8s deployment tool.

- Ubuntu 18.04 for Arm64.

Kubespray is a very good tool to ease your pain deploying highly available K8s clusters. It uses ansible playbooks and works with some major linux distributions, hardward platforms and cloud providers.

In the pre-installation you can configure your desired cluster structure, numer of masters and etcd replicated databases.

I tried some combinations of linux distributions and mostly all failed due it's difficult to find a linux distro that suits Kubespray requirements and the new Raspberry Pi 4 (4Gb RAM) hardware.
At this moment Raspbian it's compiled for Arm32/armhf and Kubespray will fail installing packages like calico, etcd has experimental support for arm32, so better to use arm64. On the other side, Raspbian Arm64 is experimental and not officially supported by Kubespray so better not to use any of these distros yet, avoiding unexpected issues.

The current latest Ubuntu 19.10 (Eoan Ermine) has a Preinstalled server image for the new Raspberry Pi 4 compiled for arm64 but it's not supported officially by Kubespray and the installation can fail so, maybe this distro will be a good candidate in near months.

I wanted a long term support stable distro to avoid collateral issues I thought about Ubuntu 18.04 LTS which will be supported for 5 years until April 2023. But, the current official Ubuntu 18.04 LTS doesn't have a preinstalled server image for the new Raspberry Pi 4, maybe they'll release it later, but now only Raspberry Pi 2/3 are supported.

While Ubuntu releases 18.04 LTS for Raspberry Pi 4 (Arm64) I found an unofficial preinstalled server image for Raspberry Pi 4, it works like a charm thanks to James Chchambers work:

https://github.com/TheRemote/Ubuntu-Server-raspi4-unofficial

Using this unofficial distro you'll still be able to update Ubuntu linux packages from the Official repo, the only thing you'll have to care is about kernel packages, but how to block them will be explained below.

Setting up Raspberry Pi 4 nodes

First let's install Ubuntu 18.04 LTS for (Arm64) in all your Raspberry Pi 4. In my case I installed it in ten Raspberry Pi 4, for a multi-master installation the recommended minimum size is four raspberry pis (as shown in the picture above): two masters, two workers (three etcds, two in master nodes one in a worker node).

I'm going to suppose that your network range is 192.168.0.1/24, your raspberry ips are 192.168.0.10-13, and your gateway is 192.168.0.1, change them accordingly with your values in the scripts below.

1. Flash ubuntu 18.04 from https://github.com/TheRemote/Ubuntu-Server-raspi4-unofficial/releases/ using balena Etcher or any other similar "flash to sd" app to all your pi nodes.

2. Boot your raspberry pi node with ubuntu 18.04 for the first time and change the default ubuntu/ubuntu user admin password.

3. If you are in the United Kingdom, change your keyboard layout to gb:

sudo vi /etc/default/keyboard

change XKBLAYOUT line to: XKBLAYOUT="gb"

4. Update all packages

sudo apt-get update -y && sudo apt-get --with-new-pkgs upgrade -y

5. Change your raspberry pi ip to static editing /etc/netplan/50-cloud-init.yaml , it should look like this one (changing ip address and gateway to your specific ones):

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eth0:
            dhcp4: false
            match:
                macaddress: dd:a6:3a:1e:50:b9
            set-name: eth0
            addresses:
                - 192.168.0.10/24
            gateway4: 192.168.0.1
            nameservers:
                addresses: [1.1.1.1, 1.0.0.1]
    version: 2

6. Block kernel packages to be upgraded from ubuntu official distribution

sudo apt-mark hold flash-kernel linux-raspi2 linux-image-raspi2 linux-headers-raspi2 linux-firmware-raspi2 linux-firmware

7. Reboot and check all changes work ok.

8. Repeat steps 1 to 7 for all raspberry pis.

Setting up a controller node

The first thing needed is a controller machine from where you're are going to install K8s to your Raspberry Pi cluster. I used my Windows 10 machine installing the Ubuntu 18.04 subsystem.

Once you have a controller node ready you'll need to prepare it and install Kubespray on it. Open a linux terminal and follow the next steps.

1. Generate a ssh key in your controller:

ssh-keygen -t rsa

It will be saved in /home/your-username/.ssh/id_rsa

2. Copy your ssh key to all raspberry pi 4 nodes, repeat this command for all raspberry pi ips:

ssh-copy-id 192.168.0.10

You'll get:

Are you sure you want to continue connecting (yes/no)? yes

(answer your password)

3. Edit /etc/sudoers in all pi nodes and add a line after the %sudo one to avoid password questions when using ssh to pi nodes from the controller node, replace "your-controller-username" by your real controller user name:

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL
your-controller-username ALL=(ALL:ALL) NOPASSWD:ALL

4. Install python3, pip and git in your controller node:

sudo apt install python3-pip git
sudo pip3 install --upgrade pip
pip --version

5. Add a function in your .bashrc script to be able to run commands to all raspberry pis at the same time:

cd
nano .bashrc

Add this function at the end of the .bashrc file:

function picmd {                                                                                                         
   echo "pi1"                                                                                                              
   ssh 192.168.0.10 "$@"                                                                                                   
   echo "pi2"                                                                                                              
   ssh 192.168.0.11 "$@"                                                                                                   
   echo "pi3"                                                                                                              
   ssh 192.168.0.12 "$@"                                                                                                   
   echo "pi4"                                                                                                              
   ssh 192.168.0.13 "$@"                                                                                                                                                                                                    }

Apply .bashrc changes

source .bashrc

Try it from your controller node writing: picmd date, you should see all pi node responses, something like this:

ubuntu@CONTROLLER-NODE:~$ picmd date
pi1
Sat Jan 25 14:50:31 UTC 2020
pi2
Sat Jan 25 14:50:33 UTC 2020
pi3
Sat Jan 25 14:50:35 UTC 2020
pi4

You can now use picmd command to execute commands (like apt update) in all your raspberry pi nodes without logging in all of them one by one.

Installing Kubespray and the Kubernetes cluster

1. In your controller node home directory clone kubespray repo:

git clone https://github.com/kubernetes-sigs/kubespray.git

2. Enter into kubespray directory and take a look which kuberspray version are you going to install:

cd kubespray/ && git describe --tags

It's important to know which kubespray version you are installing due kubespray don't support upgrades from an older release straight to the latest release, you have to upgrade it version after version without jumping any.

4. Install kubespray requirements:

sudo pip install -r requirements.txt

5. Create a new inventory from the sample folder:

cp -rfp inventory/sample inventory/mycluster

6. Declare a IPS variable with all your raspberry cluster ips, for instance:

declare -a IPS=(192.168.0.10 192.168.0.11 192.168.0.12 92.168.0.13)

7. Generate kubespray configuration based in your node ips:

CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

8. Review the generated hosts.yml file:

nano inventory/mycluster/hosts.yml

In this file, in the 'hosts:' section, be aware that node1, node2, etc... will be the hostname of your raspberry pis based on the ips you provided. Kubespray will change them, if you prefer to name your raspberry pi hostnames differently, change them in this file. Initially, hosts.yml file should look like:

all:
  hosts:
    node1:
      ansible_host: 192.168.0.10
      ip: 192.168.0.10
      access_ip: 192.168.0.10
    node2:
      ansible_host: 192.168.0.11
      ip: 192.168.0.11
      access_ip: 192.168.0.11
    node3:
      ansible_host: 192.168.0.12
      ip: 192.168.0.12
      access_ip: 192.168.0.12
    node4:
      ansible_host: 192.168.0.13
      ip: 192.168.0.13
      access_ip: 192.168.0.13
  children:
    kube-master:
      hosts:
        node1:
        node2:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

9. Optionally you can review other generated configuration files:

nano inventory/mycluster/group_vars/all/all.yml
nano inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

10. Install your multi master Kubernetes cluster using kubespray:

ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Kubespray installation process can last several minutes or more than half an hour depending on your hardware and the configuration you set, relax and enjoy a cup of tea.

Manage your new cluster with Kubectl

Let's install kubectl in your controller node to start playing with your new cluster. Download the latest kubectl version:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

Make it an executable and move it to your bin directory:

sudo chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Copy the admin.conf file from one of the master nodes to provide kubectl configuration. In the following bash commands replace the user name "ubuntu" by your controller username and the ip "192.168.0.1" with your master node ip:

cd
ssh 192.168.0.1 sudo cp /etc/kubernetes/admin.conf /home/ubuntu/config
ssh 192.168.0.1 sudo chmod +r ~/config
scp 192.168.0.1:~/config .
mkdir .kube
mv config .kube/
ssh 192.168.0.1 sudo rm ~/config

Now you should be able to list all your nodes and kubernetes pods:

kubectl get nodes -o wide 
kubectl get pods -o wide --all-namespaces

DONE!

Classic error, number of masters and etcds databases

If you are getting an error similar like this one:

[node3 -> 192.168.0.12]: FAILED! => 
{"changed": true, 
"cmd": ["bash", "-x", 
"/usr/local/bin/etcd-scripts/make-ssl-etcd.sh", 
"-f", "/etc/ssl/etcd/openssl.conf", "-d", "/etc/ssl/etcd/ssl"], "delta": "0:00:00.007623", 
"end": "2019-12-21 13:10:13.140942", "msg": "non-zero return code", "rc": 127, "start": "2019-12-21 13:10:13.133319", 
"stderr": "bash: /usr/local/bin/etcd-scripts/make-ssl-etcd.sh: No such file or directory", 
"stderr_lines": ["bash: /usr/local/bin/etcd-scripts/make-ssl-etcd.sh: No such file or directory"], 
"stdout": "", 
"stdout_lines": []}

It's because in your hosts.yml configuration the number of masters and etcd database nodes are wrong adjust them and launch kubespray installer again.

Skip Kubespray packages upgrades

In all your raspberry pi nodes, block to upgrade packages installed by Kubespray. When you upgrade your Kubernetes Kubespray cluster to a newer version, Kubespray will be in charge of update them too. Run this command in all your raspberry pi nodes:

sudo apt-mark hold apt-transport-https aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli ipset ipvsadm libipset3 libltdl7 libpython-stdlib libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib pigz python python-apt python-minimal python2 python2-minimal python2.7 python2.7-minimal socat