Francis Augusto Medeiros-Logeay

Platform Single Sign-on DIY

Sat, 22 Nov 2025 14:30:23 +0100

What?

This is a post about how to implement Platform Single Sign-on, Apple’s framework for simplifying logins from macOS devices. It builds on the SSO Extensions, but it takes it a bit further. It is also a collection of thoughts about Platform Single Sign-on and the challenges when designing it.

Why, you ask? The reason is pretty simple: it is almost impossible to find good documentation where we can understand clearly what is it that Apple want IdPs to implement when they develop their SSO Extensions. The only exception to my impression on this is the excellent article written by Timothy Perfitt from Twocanoes on the subject. Timothy also wrote a very popular example on how to implement a simple Platform SSO server.. I don’t want to repeat what Timothy wrote on his series of articles about Platform SSO. I’d rather go a bit further and discuss ideas and design possibilities, as well as what I consider lacking.

Disclaimers

My opinions are mine and mine only, and do not by any means reflect those of my employer.

Everything written here is based on using shared keys from the Secure Enclave as the Authentication Method for the Platform SSO. Other authentication methods, such as using Passwords or smartcards are not covered.

What is Platform SSO?

Platform Single Sign-On is a macOS feature allowing Macs to seamless authenticate to an IdP when they authenticate locally on their Macs. This avoids double authentication, that is, authenticating on the Mac and authenticating to the IdP using the same or different credentials.

Implementing Platform SSO from the perspective of an IdP

The rumour goes that Platform SSO hasn’t really become popular. The only two known implementations took a few years to became available, and those are basically Microsoft’s and Okta’s. It is difficult to speculate why this happened, but I have a few theories:

Lack of MDM native support: Platform SSO (PSSO from now on) is basically IdP-centric. Besides configuring Platform SSO and having the possibility to integrate device registration with MDM’s, its implementation requires IdP-compatibility and tight cooperation between Mac admins and teams responsible for authentication/identity management;
Substantial implementation of API’s on IdPs: PSSO requires some APIs that need to be implemented on IdPs. This basically requires that every IdP has to come up with their own implementation.
Scarce documentation and examples: This is probably debatable. There is documentation on how to implement PSSO, but there is little documentation with code examples and possible pattern flows. Or, in other words, sometimes it is hard to understand what Apple is thinking or how they want IdPs to implement this.
Passkeys: One could simply ask: why go through this hassle if the IdP could simply support Passkeys and call it a day? While Platform SSO gives macOS users the best possible experience, as well as giving IdP admins good tools to manage sessions, Passkeys are almost as easy to use, without having to implement a whole set of APIs to support just macOS devices.

Nevertheless, PSSO is a great addition to any IdP who wants to offer an unbeatable user experience for macOS users. The organization I work for has (through me :)) developed a Platform Single Sign-on extension for Keycloak, an open-source IdP and IAM that is quite popular. Keycloak is incredibly extendable, and could easily be extended to support PSSO.

Requirements for IdPs and how we did it

Before we dive into what IdPs need to offer Platform SSO, it is important to distinguish an SSO Extension to a Platform SSO Extension. Both will be on the same package, but one can develop an SSO Extension without support for Platform SSO. What does an SSO Extension does? Well, it basically intercepts any call to a configurable URL (the configuration needs to be managed by an MDM) so that you can add some logic on how to authenticate the user, so that other applications/websites can reuse that authentication.

On the example provided by Twocanoes, that logic, for example, is simply saving cookies the IdP sets into the Keychain, so that they are sent back to whatever attempts to authenticate again. With PSSO we might want to do things a bit differently, but the point is that there’s no recipe for what an SSO Extension should do to authenticate the user - it needs to be implemented according to the logic of the IdP. Cookies are possibly the most common pattern here, so it makes sense to use them in this context.

On an SSO Extension, everytime an application or Safari hits a predefined endpoint, the OS calles your extension, and the beginAuthorizationmethod is called. From here on you are free to do whatever you want: present a login screen if the user isn’t authenticated, send back some cookies, etc.

But Platform SSO takes this further: it fetches tokens from the IdP on behalf of the user so that they can be used by the SSO Extension to authenticate the user.

Let’s suppose your IdP does a standard OIDC flow. Platform SSO doesn’t change that, and you can develop your SSO to cope with that OIDC flow. What Platform SSO introduces is the possibility of registering the device and the user on the IdP and fetching a token automatically whenever the user authenticates on the Mac. Then the SSO Extension can use that token as a credential for the user, instead of simply presenting a login screen. The idea is that when the abovementioned beginAuthorizationmethod is called on your SSO Extension, you inject that credential (or make it available for the IdP as a cookie, for example - I wouldn’t do that, but it is possible) into the request, and your IdP will evaluate it, the same way it evaluates a password, a MFA credential, etc.

So, what do you need to implement, basically, to provide PSSO on your IdP? Well, here’s the answer, but notice that there are many ways to Rome:

Custom endpoints (Apple doesn’t really tell you how you create these, and is not so opinionated about it):

an endpoint to register the device (called Device registration by Apple)
an endpoint to register the user (called User registration), which is basically to create some sort of credential for the user based on his key - more about keys later

Endpoints that have to conform to Apple’s specifications:

an endpoint to request a nonce value to be used during logins.
an endpoint to request tokens, which is basically an endpoint where you send a login request and obtain a login response. A login request and a login response could probably be best described as credential request and credential response, or, maybe, token request or token response. Don’t think of this as just logging in the user (you do that on the SSO extension). Here, you simply obtain credentials to log that user in later on the SSO Extension.

Besides these endpoints, you need to come up with a way to recognize these tokens when the user authenticates via the SSO extension. More on that later.

When implementing these endpoints on Keycloak, we also need a client. You need it to attach the tokens you create to the it, as well as to authenticate the user when registering the device/user. Because it will be used to authentication on a native device (and not on a backend server), It shouldn’t be confidential. Therefore, it uses PKCE with SHA256 for security reasons. As per Apple request, it needs to have the urn:apple:platformsso scope.

I also recommend adding the offline_accessscope. If you don’t, the user will be prompted for a new authentication if the tokens are expired. With offline_access, this won’t happen very often.

This client is used by the SSO extension in two ways:

to authenticate the user for device and/or user registration (but we don’t have to - this depends on how you want to associate the user and the IdP, and what checks you make to allow device registration as well. Our Keycloak extension expects a token from this client;
as a part of macOS own token retrieval process.

The client also needs a single redirect uri, which is weblogin-sso://idp-login-redirect.

On our Weblogin SSO Extension, as well as on Keycloak, we used a hardcoded name for the client, so when you create yours, name it psso. In the future, we will make this configurable.

Requirements for your PSSO Extension

Well, the PSSO Extension is basically an implementation of the ASAuthorizationProviderExtensionRegistrationHandler and its methods. The main ones are:

beginDeviceRegistration
beginUserRegistration

Their implementation is quite similar. What you want to do here is to:

fetch some keys from the Secure Enclave (their public keys, mind you)
Implement some logic for the user authentication
send them to the IdP for registration

The extension needs to be configured with a profile managed by your MDM. This profile is - but doesn’t have to be - made up from multiple payloads:

One for your SSO Extension, including the Platform Extension configuration
another for the preferences of your application. Here you can save thing you will need on the app, like the URL of your IdP, the client ID, etc.

You can actually send an array of key/value strings under ExtensionDataon your SSO Extension configuration, which might make your payload much more manageable. We will do this in the future and read configuration from there instead.

This configuration will look like this:

<plist version=«1.0»>
  <dict>
    <key>PayloadContent</key>
    <array>
      <dict>
        <key>BaseURL</key>
        <string>https://<YOURINSTANCE>/realms/<YOURREALM>/</string>
        <key>Issuer</key>
        <string>https://<YOURINSTANCE>/</string>
        <key>Audience</key>
        <string>psso</string>
        <key>ClientID</key>
        <string>psso</string>
        <key>PayloadDisplayName</key>
        <string>Weblogin SSOE</string>
        <key>PayloadIdentifier</key>
        <string>mdscentral.00A38C42-503B-4016-A86D-2186CDA5989C.no.uio.WebloginSSO.3E7FAF27-6179-46AA-B1A3-B55E08D3273D</string>
        <key>PayloadOrganization</key>
        <string></string>
        <key>PayloadType</key>
        <string>no.uio.WebloginSSO.ssoe</string>
        <key>PayloadUUID</key>
        <string>3F7FDF27-6179-46AA-B1A3-B55E08D3273D</string>
        <key>PayloadVersion</key>
        <integer>1</integer>
      </dict>
      <dict>
        <key>PayloadDisplayName</key>
        <string>Weblogin Platform SSO</string>
        <key>PayloadIdentifier</key>
        <string>mdscentral.00A38C42-503B-4016-A86D-2186CDA5989C</string>
        <key>PayloadOrganization</key>
        <string></string>
        <key>PayloadScope</key>
        <string>System</string>
        <key>PayloadType</key>
        <string>Configuration</string>
        <key>PayloadUUID</key>
        <string>851A1B46-6A8A-442B-91CB-BC12FF416766</string>
        <key>PayloadVersion</key>
        <integer>1</integer>
      </dict>
      <dict>
        <key>AuthenticationMethod</key>
        <string>UserSecureEnclaveKey</string>
        <key>ExtensionIdentifier</key>
        <string>no.uio.WebloginSSO.ssoe</string>
        <key>PayloadDisplayName</key>
        <string>Weblogin SSO</string>
        <key>PayloadIdentifier</key>
        <string>com.apple.extensiblesso.CA351D35-96B1-41CF-B25B-DF3273189AAD</string>
        <key>PayloadOrganization</key>
        <string></string>
        <key>PayloadType</key>
        <string>com.apple.extensiblesso</string>
        <key>PayloadUUID</key>
        <string>4B7148CD-1069-4140-95CE-78F61BCD9C2B</string>
        <key>PayloadVersion</key>
        <integer>1</integer>
        <key>URLs</key>
        <array>
          <string>https://<YOURINSTANCE>/realms/<YOURREALM>/protocol/</string>
          <string>https://YOURINSTANCE/realms/<YOURREALM>/psso</string>
        </array>
        <key>PlatformSSO</key>
        <dict>
          <key>AccountDisplayName</key>
          <string>Universitet i Oslo - Weblogin</string>
          <key>AuthenticationMethod</key>
          <string>UserSecureEnclaveKey</string>
          <key>EnableAuthorization</key>
          <true />
          <key>EnableCreateUserAtLogin</key>
          <true />
          <key>NewUserAuthorizationMode</key>
          <string>Groups</string>
          <key>UseSharedDeviceKeys</key>
          <true />
          <key>UserAuthorizationMode</key>
          <string>Groups</string>
          <key>AllowDeviceIdentifiersInAttestation</key>
          <true />
        </dict>
        <key>TeamIdentifier</key>
        <string>YOURTEAM</string>
        <key>Type</key>
        <string>Redirect</string>
      </dict>
    </array>
    <key>PayloadDescription</key>
    <string></string>
    <key>PayloadDisplayName</key>
    <string>Weblogin Platform SSO test/V_41</string>
    <key>PayloadIdentifier</key>
    <string>37f5c3b4-36c6-101f-9485-90082e154a1a</string>
    <key>PayloadOrganization</key>
    <string></string>
    <key>PayloadRemovalDisallowed</key>
    <false />
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>dbacb344-7490-4948-b51a-b395d948fd54_41</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
    <key>PayloadScope</key>
    <string>System</string>
  </dict>
</plist>
 

How we did it on the IdP

So, I said already that Keycloak is easy to expand, right? So, what we did at first was to create the necessary endpoints. You can see this implementation on our repo on github. Note that this Keycloak extension still needs a few things to be production grade, and we’ll try to point out here what is missing. All our endpoints are configured as a Keycloak resource. You can see all of them here.

You are free to give the endpoints any name you wish. On your PSSO Extension, you need to configure the token and the nonce endpoint, as well as the keys endpoint.

Notice that, for those endpoints where Apple requires a certain standard, you need to accept requests with a few characteristics. Fortunately, Apple tells you how requests should be formed. You can check the documentation, but it is easy to see the format using this command on your terminal: app-sso platform -m, when developing your PSSO Extension.

The `nonce` endpoint:

Apple requires the nonce endpoint so that replay attacks can be avoided. So you need to implement some mechanism to receive these requests and return a value.

How does the Mac send its nonce request? According to the documentation:

POST /oauth2/token HTTP/1.1
Host: auth.example.com
Accept: application/json
Content-Type: application/x-www-form-urlencoded
client-request-id: DCAB01D3-B1FE-4E1C-802F-B3EBDCDF9E67
grant_type=srv_challenge 

Send back a json containing a key with the nonce value. You can configure the name of that key on your PSSO Extension, otherwise nonceis used.

On our implementation, the endpoint is called /nonce.

The device registration endpoint

Here, you are free to do as you want. What do you want to do here?

Basically, you want to receive the request with the device keys and persist them somewhere. You might also want to perform some sort of authentication., otherwise anyone could simple send certificates to your server.

Apple doesn’t really care how and if you do anything here. Their documentation gives some hints of what you should be doing, but they don’t dive deep into this. They say you might want to use a RegistrationToken, which is something the MDM generates dynamically so that the IdP can use to actually check with the MDM if that device is legit, or you can use device attestation. Since our MDM (Workspace ONE) doesn’t really implement theRegistrationToken`on its SSO Extension profile, we need to do it the hard way and implement device attestation. more on that later.

Our endpoint for device registration is called /enroll, and it accepts POSTrequests with a json with the following keys/values:

DeviceSigningKey
DeviceEncryptionKey
SignKeyID
EncKeyID
attestation
nonce
accessToken

The DeviceSigningKeyand the DeviceEncryptionKey are used to, well, sign and encrypt login requests and responses between the macOS device and the IdP. Their ID counterparts are used so that you can search for the keys on your database.

The attestationis a cryptographic token that is signed with the private key of your SigningKey (you can use another key here as well) that lives on your Secure Enclave. You can then check this against Apple’s root CA, which we include with our extension. You need toAllowDeviceIdentifiersInAttestation on your configuration profile so that you can extract the serial number and deviceUDIDfrom the attestation. This is information that you can use as part of your device management workflows. Our extension requires this.

On our Weblogin SSO Extension, you can see how we generated the keys and the attestation on our registerDevice()method on this file.

You need to send a nonce that you previously acquired via the /nonce endpoint.

You also need to send authenticate the user and send their accessToken. You can also see on our extension how we ask the user to authenticate. You don’t really need to authenticate the user here, actually. But since the extension doesn’t check, after the attestation, if that device is “ours” with the MDM, we introduce this authentication - which is something you need to perform for the user anyway when doing the user registration. Attestation tells us the device is legit, and that it is managed. But it doesn’t prove it is managed by us.

If you modify the extension, you might remove the accessToken verification and introduce some MDM check.

This was the only part of this Keycloak extension where we needed to use database storage. Fortunately, this wasn’t super hard to do with Keycloak.

The user registration endpoint

This endpoint is called /enrolluser and is very similar to the device registration. The keys we send are similar:

nonce
userKey
userKeyId
attestation
accessToken

What is super cool here is that Keycloak makes it very easy to save the user’s key as a credential. This allows both admins and users to revoke it on their admin and user GUI, respectively:

Keycloak stores this as a CredentialModel. Here we do need the accessToken so that we can confirm the user registering the device really has an account. Again, on our companion SSO Extension you can see how we provide the keys and attestation.

The token endpoint

Next, we created the /token endpoint in order to receive the login request and send back the login response.

We created some classes to validate the request as suggested by Apple, and also to build the response in a format the macOS will accept.

So, the endpoints above is basically what you need to process Platform SSO requests from a Mac. While the /enroll and /enrolluser are called by your extension when you decide that they should be called, the /nonce and /token endpoints need to be written according to Apple specifications.

When does the PSSO extension call these endpoints?

So, as I said, you decide when to call the /enroll and /enrolluser endpoints. You might want to call them from your beginDeviceRegistration and beginUserRegistration respectively. Everytime you start or repair your device registration, the first method is called, and when you register the user - either right after a device registration or later - the second method is called.

The /token method is called:

when the user authenticates on his mac, by restarting the machine or unlocking it
by itself when the tokens have expired

The login response will include the id_tokenand a refresh_token, and here goes a very special rant from me:

When using an authentication method that is not the Secure Enclave key, the Platform SSO will call the /token endpoint and send the refresh_token in order to get new, fresh id_tokens. But for some reason I don’t understand, it won’t do that with the Secure Enclave. Here is Apple’s explanation for that:

A refresh request uses the previous refresh token to request a new token without prompting the user for credentials. The system attempts it when the existing token hasn’t expired and the time since the last full login hasn’t exceeded the LoginFrequency in the Device Management profile. It doesn’t apply to User Secure Enclave key authentication, because the user isn’t prompted for credentials.

I really don’t get it. What is Apple trying to say here?

Should I prompt the user for credentials with the Secure Enclave when the refresh token expires?
Should we implement some logic on the PSSO extension to update that id_token?

I really don’t understand why Apple renews the id token for other authentication methods except for the Secure Enclave.

This means that we need to either:

renew the id_token ourselves, or
disregard the id_token and simply use the refresh_token as an opaque credential.

Grudgingly, we went for the second. We don’t use the refresh_token to refresh anything. We just piggyback on its verifiability on Keycloak, as well as on its long lifetime. If Keycloak is configured to only allow the refresh token to be used once, this extension doesn’t work so well. Luckily, the default configuration of Keycloak is that the reuse of refresh tokens is allowed.

We might need to revisit this in the future, but we really hope that Apple extends the automatic renewal of refresh tokens to Secure Enclave authentication. It makes more sense to use id tokens for authentication.

The authentication itself

One thing that you need to keep in mind:

In common with the Platform SSO and the SSO Extensions is the loginManager, an instance of the ASAuthorizationProviderExtensionLoginManager protocol. The loginManager has access to:

the loginConfiguration, which is where the data regarding your idp, its endpoints, etc, is saved,
the userLoginConfiguration, where you can save the username and other claims you need on each login request.
the ssoTokens - this is where you fetch the tokens you need on your beginAuthorization method of your SSO extension.

So, when the PSSO fetches the login response, it saves the id_tokenand the refresh_token on the ssoTokens member of the loginManager. That’s where you fetch them if you need them to authenticate the user at the IdP.

We developed an authenticator, which is how Keycloak calls the diverse methods to verify a user. Keycloak comes with built-in authenticators, such as username/password, OTP, kerberos, passkeys, etc., and allows developers to code their own.

Our authenticator inspects the header of the authentication requests and checks if a Platform-SSO-Authorization header is present. If so, we evaluate it and authenticate the user.

The token we send with the header is an encoded json in base64 format, with the following key/values:

refresh_token
kid (the Signing Key ID of the device, so that we find which key to use for verifying that this is a legit request)
signed_at
username (doesn’t really work, it seems Apple doesn’t allow the explicit use of the saved loginUserName

Since we added the username to the refresh token, with this data the authenticator will be able to:

check if the request came from a known device
validate the refresh token using Keycloak’s own API
consider the user authenticated
attach all authentication results from the same device to the same session, which makes it easier to manage sessions (and the reason why we’d love Apple to allow automatic renewal of id tokens.

So our SSO Extension basically does this:

when triggered by a call to the /protocol/openid-connect/auth, the SSO extension injects the header with our token, as described above,
if the response is not the callback with the value from the redirect_uri with a code, but rather a form with a password field, we display the login window.
we return the the browser as soon as we get a redirect to the callback, which is always the redirect_uri.

It should work a bit the same way with SAML, except it doesn’t. So, right now, our SAML flow is a bit erratic. If you can help us to fix it, we’d love to hear from you.

We don’t care about the cookies anymore, because if another application needs the cookies, they simply authenticate again and the extension will give them back with the response. The SSO is performed by injecting the token into the request, not by keeping cookies. All in all, it will be the same session anyway.

A few things that don’t work well yet

There are a few things that need to be fixed:

better handling of required actions: Keycloak has some required actions that, when called from their internal clients like the Account console, seems to create a reauthentication flow that don’t play ball well with our interception of the authentication url. It works perfectly now, but the required action is performed inside the SSO Extension, not on the browser. We’d like to get this done on the browser, but there’s a conflict with cookies that we can’t seem to solve.
SAML, as pointed above,
the implementation of some security checks during the authentication, the same way that Keycloak does when using their CookieAuthenticator. This will be done soon, but until then, if your Keycloak instance makes use of ACR/LoA, this authenticator might not comply with your authentication rules.

Conclusion

We believe that this implementation might be helpful for a lot of people that want either to try Platform SSO or to provide Platform SSO without having to rely on one of the big IdPs. Theoretically, with a few modifications and by using Token Exchange, , this solution can potentially be used in a way where Keycloak becomes a broker between Macs and other IdPs, but this is not something we tested or implemented.

It would be very nice if other developers could join our efforts, especially when it comes to the SSO Extension and its processing of SAML flows. If you can and want to help, send PR’s our way or drop as a line on the #Keycloak channel at the MacAdmins Slack .

Finally, I just wish Apple could be a bit more explicit on how they believe this extension should be used:

After a token expiration, should we renew automatically for the user (after all, the loginManager has a method for that, or should the user authenticate manually?
How should we handle SAML flows?
Why no refreshing of tokens for Secure Enclave flows?

I think there’s a lot that could be accomplished here if some of Apple’s intentions were known. But i must admit that, after implementing this extension, a lot of the documentation makes more sense - in the beginning, it felt insufficient, but I guess that’s mostly because there are no examples or design patterns shown anywhere, except by those examples from Twocanoes.

Acknowledgments

We are very grateful to the work of Timothy Perfitt and Joel Rennich, who across presentations and articles made this subject a bit clearer to a lot of people, myself included.

I am also grateful to my colleagues Gaute and Thomas, who encouraged me to write this, and who came with good ideas and feedback along the way.

How to use Keycloak as an IdP for Entra ID via ADFS

Wed, 11 Jun 2025 22:30:23 +0200

What?

This is a post about how to use Keycloak (or another SAML-supporting IdP) as an IdP for Entra ID via ADFS. Or, putting it in more technically correct terms, how to use Keycloak as an IdP for an ADFS instance that is federated with Entra ID.

I decided to write this after suffering a lot to find good documentation and/or reports on how this should be done.

Disclaimers

I think it is fair to say a few things before we get started:

I don’t like Microsoft. You’ll see a bit why when you keep reading this document.
I am not, and never have been, a Windows user. So my experience on Windows is quite limited
This document represents my own opinions and ideology, and not those of my employer.

Thanks to my awesome Windows colleagues who helped me testing a few of these things that I describe here.

Why?

I am glad you ask.

You see, there are many reasons why you’d want to federate your authentication when using Microsoft:

You might have your on-premise Keycloak or another IdP, but you also have Microsoft services, and users end up having two different authentication portals;
You don’t like ADFS’ own authentication portal;
You certainly don’t want to export your password hashes to Microsoft, especially in these Trump-times.

The good thing is that by going this route, your users will have single-sign on between your Microsoft and their apps (including those you register on Entra ID) and your Keycloak-based authentication provider and apps that use its instance for authentication.

The bad thing is that you need to choose what to do with 2FA. Will you keep it on Microsoft? Will you use Keycloak’s 2FA options? Either way, there are options we will discuss.

Introduction

Suppose your organization uses Entra ID, and it isn’t ready to sync your users’ passwords to your Entra/Azure tenant. You still want to provide on-prem authentication.

Entra ID allows this in what is called “federated” authentication. But this is Microsoft: they simply are unable to use their definitions in a consistent way. If you google “federated authentication microsoft”, you will see that what they call “federated authentication” can be two things:

Having authentication delegated to an external IdP, as described here.
Having an external IdP configured for external users, ie, not your own users, as described here.

We are interested in authenticate our own users on our premises. This is something Microsoft does not document well and they don’t really go out of their way to help with this, since they really want you to export your password hashes to the cloud. But it is supported nevertheless.

So you have two (actually three) options here if you want to use on-premise authentication:

use the good’ol ADFS;
use Keycloak directly, by configuring Keycloak as a SAML federated IdP for Entra ID;
use the good’ol ADFS, but configuring it to use Keycloak as its IdP.

Why don’t you just federate with Keycloak directly? Well, because Microsoft. They list some limitations on the document about how to configure a SAML IdP, but the real deal is if you use Intune with Entra ID, device registration won’t work if the IdP you use does not support the protocol who is dying but Microsoft keeps it breathing: WS-Fed. You can read about this limitation here, but the problem is that nobody, including Keycloak, supports WS-Fed.

But not everything is lost: you can still use ADFS (which also refuses to die), and configure it to redirect your users to your on-premises IdP - in our case, Keycloak. So while you won’t get rid of ADFS, it will give you the compatibility you need.

Configuring ADFS to use Keycloak as an IdP

Here we need to clarify a few definitions, because Microsoft refuses to call things like everyone else.

Look at this table:

Microsoft	SAML	OAuth2
Claims Provider Trust	SSO IdP	IdP/Resource server/
Relying Party Trust	SSO SP (Service Provider)	client

So here what you need is to configure Keycloak as a Claims Provider Trust (CPT) on your ADFS. A Claims Provider Trust is basically an IdP. The default CPT in ADFS is Active Directory, but you can configure a SAML (or a WS-Fed) IdP there as well.

A Relying Party Trust (RPT) is an application that authenticates via your ADFS. In other words, they authenticate on one (or more) of your CPT’s. In SAML lingo, they are SP (Service Providers). In the scenario we are talking about here, where Entra ID will send users to its federated authentication IdP, the RPT is… Entra ID! But you probably know that by now: when you use Entra Connect to configure User sign in so that users will authenticate on ADFS, it creates an RPT that points to Entra ID on your ADFS.

So, without further ado, how do you add Keycloak as an IDP?

open ADFS, click on Claims Provider Trust on your left, and then Add Claims Provider Trust on your right

On the window that shows up, choose “Start”
Enter the SAML metadata for your keycloak instance and realm, something like https:///realms//protocol/saml/descriptor:

follow the additional steps which are self explanatory.

There, now Keycloak is an IdP. But you are far from done, my friend. This is Microsoft, and you need to get into the Microsoft way of doing things…

Claims, claims and more claims

Coming from Keycloak world, you are used to something called account matching: you can configure Keycloak in a way that, when you use an external IdP, if the username or e-mail of the external user already exists on your user base, it will then establish a link between the external and internal users, so that the internal user’s attributes are the ones that Keycloak sends to the applications. This is a way to simplify it, things can be configured differently, but the point is that a pretty default way of using keycloak is establishing links between internal and external users.

ADFS works very differently. When you configure a CPT, the user claims received from Keycloak are not used to fetch a user and all their attributes. If you want to match a Keycloak user to an Active Directory user (and you know you want that), you need to match claim by claim. You cannot simply say “Oh, fetch me the user myuser@mydomain.com and their attributes”. You need, in a way, reconstruct the claims your RPT needs.

When configuring Keycloak to be used as an IdP for ADFS on an Entra ID scenario, you will see that Entra ID expects a lot of claims in a very special format, something like that http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress. So you need to provide a claim like that.

Let’s see how you do that:

Add Claim rules to your newly created CPT

Click on “Add rule” on the window that shows up.
Choose “Send claims Using a Custom Rule”:

I am using custom rules here because I need to translate the claims from Keycloak to those used by Entra ID. I could use other types of rules (like for example “passthrough” if the claims already come with the right claim name from Keycloak. This is really up to you.

You’ll get a window like this:

Here you need to enter something in a Microsoft ADFS Claim language. Basically what you want is to say:

Which claim received from Keycloak you want to do something about;
How that claim will be called (and made available to) the RPT’s.

Let’s start with the userPrincipalNameclaim. Entra ID requires the so-called UPN in a few formats. So if you have a claim in keycloak that gets your user in the user@domainformat, you fetch it, and give it a name Entra ID will like. For example:

c:[Type == "userPrincipalName"]
 => issue(store = "Active Directory", types = ("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn",
"http://schemas.xmlsoap.org/claims/UPN"), 
query = "(userPrincipalName={0});mail,givenName,sn,userPrincipalName,userPrincipalName;MYDOMAIN\adfs", param = c.Value);

Ok, this is confusing. What happens here?

I get a claim from Keycloak called userPrincipalName
I query that claim in Active Directory to find a user that matches it
I assign this user’s mailgivenName, sn and userPrincipalName to the clames on the “types” array.

Of course, you can simply send the claims with values you fetch from Keycloak, this is fine. But I’d rather use Active Directory as an authoritative source for the user’s identity.

The MYDOMAIN\adfs is something not clear to me. It seems that you put any user here, and it will work - as a sort of placeholder. But in some documents I read this needs to be an existing user, specifically one that binds to AD. I admit I am not sure, so I just used one I created when configuring ADFS. Again, Microsoft’s documentation is not very nice here.

You still need to add two more rules. After saving the first one, add another rule, like the first one, but with the following content:  

c:[Type == "userPrincipalName"]
 => issue(store = "Active Directory", types = ("http://schemas.microsoft.com/LiveID/Federation/2008/05/ImmutableID"), query = "(userPrincipalName={0});ms-DS-ConsistencyGuid;MYDOMAIN\adfs", param = c.Value);

Again you use the userPrincipalNameto fetch an attribute that will be used as the ImmutableID.

The last mandatory rule is this one:

c:[Type == "userPrincipalName"]
 => issue(Type = "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Value = "MYDOMAIN\" + regexreplace(c.Value, "^(.*?)@.*$", "$1"));

You are good to go! Now you’re ready to test it. Go to https://login.microsoftonline.com, enter your username, and you will be redirect to your Keycloak instance. Or maybe you will see a list with your Keycloak instance name and Active Directory. This is fine for testing, but I guess you want the user to be sent right away to Keycloak.

If so, use the following command:

Set-AdfsProperties -EnableLocalAuthenticationTypes $false

I configured way more claims on my test instance. That’s because I see that Entra ID, on its RPT’s Claims Issuance Policy, uses a few other claims that Active Directory has, but Keycloak doesn’t, such as primary group, objectguid, msdsconsitencyguid, etc. But to be honest Ii am not sure you need all those. At the end of this articlel I give you a list of all the rules I configured, and you can decide if you want to configure them or not.

2FA or not 2FA? That’s the question

Now, suppose you have 2FA configured in Keycloak, or in Entra, or in both. You need an strategy here: are you going to use Keycloak’s, Microsoft Entra’s, or both? Both is not ideal, of course.

The issue here is that you might want to give up Microsoft’s 2FA. For security reasons, you might not want to turn off Keycloak’s 2FA for the ADFS client. You want to make sure that a Keycloak user who authenticated via Keycloak has used 2FA.

The problem is this: suppose you turn off 2FA in Keycloak for the ADFS authentication, relying on Entra ID’s 2FA. The user will authenticate in Keycloak, but just with username and password. The same user don’t go ahead with Entra ID authentication. But he is already fully authenticated in Keycloak! So if he goes ahead to another application, he’ll be already logged in, thus skipping 2FA.

So that’s why you want to keep 2FA in Keycloak, but, in such integration you might want to send a message to Entra saying “Never mind, this user has already used 2FA with me, let him without further ado”.

This is something where the documentation fails blatantly. So what you have to do is to add a Claim Issuance Policy to your RPT (in this case, Entra’s RPT, usually called “Microsoft Office 365 Identity Platform Worldwide”, and add a rule there:

Then choose “Add rule”, “Sending claims using a custom rule”, and enter the following rule:

=> issue(Type = "http://schemas.microsoft.com/claims/authnmethodsreferences", Value = "http://schemas.microsoft.com/claims/multipleauthn");

Please notice that this will also make Entra ID stop asking for 2FA even if you actually authenticate from Active Directory. To avoid that, don’t configure the rule above. Instesd, configure it on the CPT, and add it again to the RPT, like this:

CPT:

=> issue(Type = "http://schemas.microsoft.com/claims/authnmethodsreferences", Value = "http://schemas.microsoft.com/claims/multipleauthn");

RPT:

c:[Type == "http://schemas.microsoft.com/claims/authnmethodsreferences", Issuer = "mykeycloakcpt"]
 => issue(claim = c);

Remember to replace “mykeycloakcpt” for the name of your CPT that has your Keycloak configuration.

This was just another example how Microsoft’s documentation can be misleading. The documenttaion found here never worked. It seems the claims were all wrong. It was only after seing how Ping does it that we got it right: See here.

Oh, and run these Powershell commands:

Run these two commands on Powershell:

Update-MgDomainFederationConfiguration -DomainId <yourdomain> -InternalDomainFederationId  <yourdomainfederationid> -PromptLoginBehavior nativeSupport

and

Update-MgDomainFederationConfiguration -DomainId <yourdomain> -InternalDomainFederationId  <yourdomainfederationid> -federatedIdpMfaBehavior acceptIfMfaDoneByFederatedIdp 

Conclusion

I hope this helps you to configure Keycloak as an IdP for Entra ID. It’s recommended to test this with a lot of authentication scenarioes, like e-mail, device enrollment, etc, to see if everything works.

Appendix

Here is a list of other CPT rules I configured, though I’m almost sure we din’t need them:

[](msdsconsitencyguid:

c:[Type == "userPrincipalName"]
 => issue(store = "Active Directory",
          types = ("http://schemas.microsoft.com/ws/2016/02/identity/claims/msdsconsistencyguid"),
          query = "(userPrincipalName={0});ms-DS-ConsistencyGuid;MYDOMAIN\adfs",
          param = c.Value);

objectguid:

c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"]
 => issue(store = "Active Directory",
          types = ("http://schemas.microsoft.com/ws/2008/06/identity/claims/objectguid"),
          query = "(userPrincipalName={0});objectGUID;MYDOMAIN\adfs",
          param = c.Value);

groupsid:

c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"]
 => issue(store = "Active Directory",
          types = ("http://schemas.microsoft.com/ws/2008/06/identity/claims/groupsid"),
          query = "(userPrincipalName={0});tokenGroupsSid;MYDOMAIN\adfs",
          param = c.Value);

Account type:

c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"]
 => issue(store = "Active Directory",
          types = ("http://schemas.microsoft.com/ws/2012/01/accounttype"),
          query = "(userPrincipalName={0});objectClass;MYDOMAIN\adfs",
          param = c.Value);

onpremobjectguid:

c:[Type == "userPrincipalName"]
 => issue(store = "Active Directory", types = ("http://schemas.microsoft.com/identity/claims/onpremobjectguid"), query = "(userPrincipalName={0});objectGUID;MYDOMAIN\adfs", param = c.Value);

primarysid:

c:[Type == "userPrincipalName"] => issue(store = "Active Directory", types = ("http://schemas.microsoft.com/ws/2008/06/identity/claims/primarysid"), query = "(userPrincipalName={0});objectSid;MYDOMAIN\adfs", param = c.Value);)

E-mail authentication, quo vadis? (Or where is OIDC/Oauth2 for everyone?)

Sun, 02 Feb 2025 10:15:23 +0100

There’s something rotten in the world of e-mail.

You see, if we come back twenty-five years ago, e-mail was handled in-house by companies or by your local ISP. People did have their hotmail or yahoo accounts, but you’d always get one from your ISP.

Then companies like Google started to host e-mail for companies, but it was still possible to run your own e-mail server. Heck, it is still is.

But a certain challenge came in: multi-factor authentication. Today, MFA or 2FA are absolute requirements for any authentication, and e-mail is not an exception.

Suddenly, it got hard to use plain username/password login for IMAP and SMTP, the most widely used e-mail protocols if the goal was to have 2FA. The alternatives would be adding 2FA to a password, but that would prompt the user every now and then to reenter the password with the OTP.

That’s why over the last few years the big providers moved for OAuth2 authentication: the user credentials in form of username/password aren’t sent to the e-mail server anymore: the user is redirect to a browser, then he authenticates on his IdP (Identity Provider), get a token which is then sent to the e-mail server.

Luckily, some of the most popular e-mail servers do support OAuth2. But unluckly, nome of them allow you to add the e-mail server you want. What happened here is that e-mails usually come with a pre-configured list of OAuth2 providers (Google, Microsoft, Yahoo, etc). This is strenghtening e-mail as provided by the Big One’s and preventing anyone to have an e-mail server with modern authentication.

This is partially a problem due how Oauth2 works: it requires an IdP to configure a client, and e-mail clients need information about that client to talk with them - often with a client secret.

So far, no e-mail client that I know of allow you to simple enter the OAuth2 configuration of your IdP of choice. The Big One’s have their credentials baked in the different e-mail clients. See, for example, this source code from Thunderbird.

Just for fun, I tried to bake my own IdP on the source code of Thunderbird. And it worked very well! And it begs the question: why can’t people add their arbitrary IdP on e-mail configurations? If you can choose the authentication method between username-and-password, kerberos, certificates, etc, why can’t you choose OAuth2 and configure the clients accordingly?

To be clear, this is not so much of a problem for webmail. Many webmail solutions like Roundcube support OAuth2. I configured it to use OAuth2 against my Dovecot server, and boom, I get single sign-on between all my applications. But saddly, no e-mail client seems to allow adding OAuth2 as an authentication type. Either it comes pre-configured by default for the Big Ones, or you can’t add it.

Unfortunately, it doesn’t seem to be getting any better, by judging from this five year long thread about implementing OAuth2 on Thunderbird. If an open-source client, which should attempt to serve the open-source cause, isn’t supporting it, why should those default clients tied to the Big Ones?

What happens? The Enterprise world is led to believe that they need to get e-mail from the Big Ones, as nobody seems to be able to give them 2FA for e-mail. E-mail providers’ diversity shrinks.

I loved the work of Michael W Lucas, especially his Run Your Own Mail Server book. We need to get e-mail diversity back. But the world is moving on. New protocols such as Jmap start to get attention. Still, modern authentication isn’t available for everyone on the e-mail world, unless your e-mail is hosted by the big ones.

This has to change.

The complexities of running a Fediverse instance on clusters

Sun, 19 Jan 2025 17:01:23 +0100

Disclaimer

I am no expert on Kubernetes and neither have I a deeper understand of all the components of the different Fediverse servers. What I am writing here is a result of my own impressions, which may be incorrect or misleading.

Introduction

I started to run Fediverse instances in november 2022. Our Mastodon instance, Babb.no has been running since. Since then, I also started to run a Pixelfed instance, a Bookwyrm instance and finally a Friendica instance.

My goal was to offer colleagues and friends a gateway to healthier social networks. Well, those didn’t come, so I opened these networks to everyone.

Since I don’t have exaactly so much free time to understand these services and fix or scale them when the need that, it is important for me that I can run those services on a kubernetes cluster, to have a nice forum to exchange information on how to fix things, and to have good documentation.

Sadly, these services are not always easy to understand. They have sometimes a lot of components and it is not trivial to understand what they do nor how they relate to each other.

I want to compare how these instances in three areas:

cluster deployability
documentation
support community for sysadmins

Cluster deployability

These days, enterprise environments for web services are hosted on cluster like Kubernetes.

However, because Fediverse instances are primarily run by individuals with limited access to resources, I suspect that there are few tutorials, if at all, on how to run Fediverse instances on a cluster.

Among all the servers, things vary considerably. From having a helm chart, to not even mentioning clusters at all.

The main problem here is that there is very little documentation on which components can or need to be deployed separately so that one can use cluster for redundancy and high availability.

Mastodon is clearly the winner here. It provides Helm charts, so one can have things deployed quite easily. However, I didn’t go that route. I used their Docker compose file, read a lot of posts about mastodon sidekiq workers, and created my yaml files using Kompose. It gave lots of flexibility back in the days, though today I’d probably force myself to understand Helm.

Pixelfed didn’t have an official Docker image back in the days, so I got a Dockerfile somewhere, and deployed it myself. Luckly, its design is simple - one needs a web container and a worker container. Both can scale, and it works fine.

Bookwyrm wasn’t so easy either, to the point I created my own repo with configuration files.

The problem with Bookwyrm, and as well as with Friendica, is that while they do provide ready-made docker images, they are based on the premise of a single node deployment. So we end up having a redundant Nginx container, and on neither of those we have a clear idea on a good deployment architecture.

A little digression here: people deploy clusters differently when it comes to web access: some have an ingress (Nginx) exposed to the world, some have a reverse proxy pointing to services, some (like me) have a reverse proxy that proxy requests to an ingress.

So a clear documentation on what is the best way to implement routes, block paths, etc, would be great for cluster maintainers.

Bookwyrm is probably scalable - I never tried to increase the number of its workers, but I see no reason why it wouldn’t work.

Friendica was the hardest due its confusing docker documentation, no good php-fpm tweaks (like for example fixing pm.max_children needs to be done manually) and no scalability whatsoever. I guess that at some point I’ll have to limit the number of new users. One may most likely increase the Nginx pods, but there is no worker scalability.

Documentation

While none of those servers have cluster-specific documentation, they have some documentation of configuration items.

Mastodon is again the winner. With a mature structure and development, it has an up-to-date documentation on all configuration options, though some are not explained on the list of configuration parameters. For example, PAPERCLIP_ROOT_URI is listed as valid configuration, but you’d have no idea of what it is used for unless you read its Object storage configuration (even though this applies to local, filesystem storage).

Bookwyrm has a good installation, but not good index of configuration options, which is useful when converting installation patterns to a cluster.

Pixelfed was very good before, but the documentation is so outdated these days that some configuration is only found on the source code. Fortunately, the main developer is accessible and usually helps right away.

Friendica has documentation on different sites, which is confusing, as you don’t know what’s current and what’s old. It’s a wiki and documentation site. This makes find things difficult

Community and help for sysadmins

If you’re a Patreon to Mastodon, you get access to their Discord. There, there’s a channel for sysadmins (I miss the term “sysops”), and one gets a lot of help. It has saved me many times, so I feel a bit safer knowing I’m not on my own.

Pixelfed also has a Discord and a channel there for sysadmins. However, help, true help, comes only from it’s main (only?) writer. So one’s milleage may vary: if you need help when it’s night in Canada, you’ll have to wait. Unfortunately, Pixelfed is highly dependant on one developer, and it’s impressive that, in spite of this dependancy, things go well and one does get help. It just might take a bit of time.

Bookwyrm has a Matrix room for admins, but seldom one gets help there. Sometimes posting on their github issues will get attention, but it might take weeks. I feel that development there is quite stalled, but I hope I’m wrong.

Friendica was very difficult for me one year ago when I tried for the first time. Nobody was running it on Kubernetes. Replies were vague, and it was hard to get help. I gave it a try again recently, and, while I haven’t anyone running it on a cluster, I did get better help this time on their group on, well, on Friendica. They do have a Matrix room, but I didn’t get too much help there.

Conclusion

I want to make clear all this is based on my own experience, and it might be based on misunderstanding on where to get help or configuration information.

That said, if you’re planning on deploying and maintaining a Fediverse instance on a cluster or any other deployment that’s different than the instance’s default install instructions, you might be on your own.

Mastodon, being much more mature than the others, have a robust community of developers and seasoned admins who almost certainly will help you.

Unfortunately, that can’t be said about the other fediverse services. I do hope that these services gain critical mass so that they’ll get more resources and make them easier to administer.

Installing Matrix on Kubernetes - what they don't tell you

Tue, 26 Nov 2024 21:39:23 +0100

Disclaimer

I am no expert on Matrix or on Kubernetes. I just happen to use Kubernetes a bit, and got recently hooked on Matrix.

This is not a complete manual or guide on how to install Matrix on kubernetes. It is rather an introduction on the challenges you might experience and a heads up so you don’t make the same mistakes I made.

Why are you writing this?

I wanted to install Matrix. I have install (and still maintain) a few opensource-based projects, specially Fediverse instances. I thought, what can’t wrong, right?

In my mind, these services follow the same design pattern, which is basically:

a web-server/streaming service
a few workers
redis
database

While Synapse (one of the implementations of the Matrix protocol) has these, its installation is challenge, mostly because:

the documentation is not exactly the most idiot-proof,
the workers do not behave like other works of, say, Mastodon,
load balancing is tricky.

Of course, if you want to do something basic, the official documentation will get you going. But one someone installs something on a cluster, the main advantage is quick scalability. And that’s the thing there are few guidelines about on the internet.

I’ll comment a few things about the last two things on that list, but first, one little note: if you don’t use Helm because, like me, you’re too lazy to learn it, do yourself a favor and learn how to use it. I got away by not using it to all the stuff I deployed (mastodon, pixelfed, bookwyrm, etc). But Matrix is a beast. It’s the kind of project you’d expect some help. I’ve seen people using Nix to maintain the deployment code, and if you speak Nix, lucky you. Someone I met on Matrix the other day send me its setup, which is a work of art.

But if you’re lazy like me, you will have a lot of yaml writing to get this done.

So, get the configuration done by following the first steps of the abovementioned official documentation, and read along.

The workers

Workers are the main unit when you want to scale up things. On kubernetes, a worker usually is a deployment, so one just scale up the number of replicas.

It is a bit the same with Matrix, but with one caveat:

Workers need to have a unique name. Some of them, like the federation_sender’s, need to be explicitly referred to on the main configuration. So you cannot simply go ahead and increate the number of workers. Or, you can, provided that they can be configured with a unique name. There might be many good strategies to do so in Helm or even without it. Mine was this:

 command: ["/bin/sh", "-c"]
          args:
           - |
             sed  "s/client_worker/${POD_NAME}/g" /config/client.yaml > /tmp/client.yaml &&
             exec python -m synapse.app.generic_worker \
             --config-path=/data/homeserver.yaml \
             --config-path=/tmp/client.yaml
          ports:
            - containerPort: 8022
              name: http
            - containerPort: 9093
          volumeMounts:
            - name: client-config
              mountPath: /config
              readOnly: true
            - name: matrix-volume-claim
              mountPath: /data
      volumes:
        - name: client-config
          configMap:
            name: client-config
        - name: matrix-volume-claim
          persistentVolumeClaim:
            claimName: matrix-volume-claim
      restartPolicy: Always

This is how I use the image to deploy a generic worker of the type client. The config map looks like this:

onfigmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: client-config
  namespace: matrix
data:
  client.yaml: |
    worker_name: client_worker
    worker_app: synapse.app.generic_worker
    worker_listeners:
      - type: http
        tls: false
        port: 8022
        resources:
          - names: [client,federation,media]
        bind_addresses: ["0.0.0.0"]
    redis:
      enabled: true
      host: 10.20.20.202

So when scaling up, it will modify the ConfigMap when creating the pod. This is great for deployments of workers that don’t need to be on the homeserver.yaml. If you need those, I suggest using StatefulSets, since their names are predictable, and add them to the homeserver.yaml. Of course, unless you automate this, you need to adjust the amount of those workers on the main configuration manually if you increase or decrease them.

Check the documentation on workers. Some must be only a single instance, others can scale up independently (as long as their name are unique), and some needs a restart of all workers and a configuration change to work.

Load balancing

On most services, load balancing is done by redirecting traffic to some sort of web workers.

On Matrix, it is more complicate than that. Load balancing here is done mostly by isolating some endpoints closely related to each other, like synchronization or federation activities, and sending them to respective workers.

But besides that, two workers need more attention: federation inbound and sync.

Sync is done in such a way that load balancing is optimal if it’s done by user. The documentation has a few tricks on how to do that. On an nginx server, this is kinda trivial. But on the nginx ingress controller, this can be a bit more complicate.

I went ahead and deployed a new ingress controller just for the matrix namespace on my cluster. That way, the rules I added won’t be applied to traffic to other services.

I then added an http-snippet with the mappings mentioned on the documentation so I could get the variables to be used by nginx to do the load balancing:

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  annotations-risk-level: Critical
  http-snippet: |
     map $arg_since $sync {
     default "matrix-sync-8022";
     '' "matrix-sync-initial-8022";
     }
     map $arg_access_token $accesstoken_from_urlparam {
     default   $arg_access_token;
     "~syt_(?<username>.*?)_.*"           $username;
     }
     map $http_authorization $mxid_localpart {
     default                              $http_authorization;
     "~Bearer syt_(?<username>.*?)_.*"    $username;
     ""                                   $accesstoken_from_urlparam;
     }
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: matrix-ingress
    app.kubernetes.io/name: matrix-ingress
    app.kubernetes.io/part-of: matrix-ingress
    app.kubernetes.io/version: 1.12.0-beta.0
  name: ingress-nginx-controller
  namespace: matrix

You see that this requires predicting how nginx is going to refer to the services you create. Here, matrix-sync-2022 is a deduction that the controller made by <namespace>-<service name>-<portname>.

That done, I used annotations on my ingress:

    nginx.ingress.kubernetes.io/upstream-hash-by: "$mxid_localpart"
    nginx.ingress.kubernetes.io/configuration-snippet: |
        if ($request_uri !~ "^/_matrix/client/(unstable/org.matrix.simplified_msc3575|v5|v4|v3|r0|v1)/sync" ) {
        set $proxy_upstream_name $sync;
        }

This allows:

sending initial synchronizations, which are heavier, to particular workers;
load balancing things by the user

The federation inbound is nicer with load balancing by ip, so you update the upstream-hash-by with the $remoteaddr (I think and hope).

Other things

There are things that, as of today, are pretty much basic on Matrix, but the documentation doesn’t mention that you’ll want to install them, probably because they are still considered “Experimental”, though they are widespread, and new clients such as “Element X” rely on those.

The first is Matrix Authentication Service, the new authentication layer based on OIDC. You’ll want that, because that’s the future.

The other is Element Call. If you don’t have it, audio/video calls won’t work.

The last thing is a turn sever, like Coturn, which is a TURN server to assist voip with other clients.

Conclusion

I really liked Matrix. It performs great if you don’t go to a room with 50.000 people. It’s installation, though, is very complex and granular. I heard that there are people that even create load balancing rules for a particular room, such as Matrix HQ, so that it won’t steal much processing.

I hope this was useful for you. And thanks to the guys on the “Matrix on kubernetes” room on the Matrix space who helped a lot to understand a few of those things.

Babb.no - The Fediverse and open social networks

Sun, 17 Dec 2023 12:39:23 +0100

Introduction

TL;DR: About babb.no.

I really want to stop using social network in this age of algorithms, monetization of personal data and AI. I like when I am in control of my own data, where I can migrate my data wherever I want.

Last year, I got to know about Mastodon, a microblogging platform that gained fame just after Elon Musk bought Twitter. I decided that Mastodon was something for me, so I started Mastodon.babb.no.

Then I discovered Pixelfed. And then Friendica. And finally Bookwyrm.

The cool thing about these sites is that:

There is no algorithm scraping my data and offering me ads to buy stuff;
no hidden use of my data and pictures;
I can focus on content rather than skipping through useless ads or content I don’t want to see,
The API’s are open, so I can do a lot of things.

The sites above work sort of an alternative to known services:

Mastodon is an alternative for Twitter/X/Threads;
Pixelfed is an alternative for Instagram;
Friendica is an anternative for Facebook;
Bookwyrm is an alternative for Goodreads.

The cool thing is that if you have a user on one of these sites, you can follow anyone on the other instances. So if I have a user on Mastodon, I can follow users on Pixelfed. And the otherway around. This is thanks to the ActivePub protocol.

They are all fully functional services, though somethings are not there yet, the most notable ones being Friendica and Bookwyrm not having an iOS app (or an app at all, like Bookwyrm).

I decided to self host these websites. I could have joined other instances, but I believe it’s good to give something back.

My instances are:

Mastodon: mastodon.babb.no
Pixelfed: pixelfed.babb.no
Friendica: social.babb.no
Boomwyrm: books.babb.no

Nerdy talk

They were all relatively easy to install, but Bookwyrm was a bit harder as the installation was really based on Docker-compose. Since I run a kubernetes cluster, I had to adapt things a bit. I posted the whole thing on Github.

Conclusion

I hope that someday we will all use social media that is community-driven, I really like social media, not less because I need a way to keep in touch with friends and family who live far. The challenge today is to get people to migrate to these opensources solutions. But now the stepping stones are in place.

Running a Mastodon instance on kubernetes

Fri, 09 Dec 2022 15:20:23 +0100

Introduction

I heard about Mastodon when it got widely mentioned as a result of Elon Musk’s Twitter acquisition. Its goal really got me interested, and I fell in love about it from day one.

I decided to run my own instance, Mastodon.babb.be, and it has been a joy to use it and to maintain it.

I want users to come, but when I heard other stories of servers having to quickly scale up in order to accomodate the huge number of users coming from Twitter, I decided to get ready for such influx should it happen.

My goal was to setup a kubernetes cluster for my static websites, as well as for my old WordPress-based blog. The cluster was installed, everything seems to be running smooth (despite Ubuntu 22.04 quirks with kubernetes), so it was time to get Mastodon moved to it.

The challenge

There were few recipes on how to move Mastodon to kubernetes.

Mastodon comes with a Helm chart, but I really didn’t want to use it - some people out there say it is quite old. I also saw some other kubernetes deployments, but they were from more than four years ago, and a lot has changed.

How

I had already a running, non containerized instance. So I had already a configured setup, which made things easier for me.

Initially, I used Kompose to convert the docker-compose.yaml supplied by Mastodon’s repo to kubernetes deployments, services, etc.

After that, some cleaning up and adaptation, I did :

throw out some files, like network configuration, redis, database and postgresql deployments, etc, as I run most of the persistent stuff outside my cluster.
create several deployments for the sidekiq queues, as suggested by Nora.
move sensitive key/values out of the env.production configMap to a secrets file
created an ingress for web and another for streaming. I probably overdid it a bit, as I have a reverse proxy outside my cluster, so I moved some of nginx configuration from mastodon to my ingress, just to be on the safe side.
finally, just for the kicks, I added a topologySpreadConstraints to my deployments to make sure the pods would be allocated evenly on my cluster (I got this one here.

I applied all the yaml and my cluster was then online.

Remained things to do:

Remember the cronjobs? Well, we need those. I need to create a CronJob resource on my cluster, but, for now, what I did was that I stopped all my mastodon services on my original instance, but kept the cronjobs there. I’ll remove those eventually and create CronJob instances on the cluster.

Please give me any comments or feedback. All the files I ended up using can be found on my GitHub repo.

Booting a Raspberry Pi 400 from FreeNAS with Unifi

Sat, 27 Mar 2021 12:20:23 +0100

Introduction

I am a hopeless collector of Raspberry Pi’s: they come out, I try to grab them. And when I saw the Raspberry Pi 400, being the nostalgic ex-Amiga user that I am, I knew I just had to buy one. My little daughter sometimes asks me to boot the Pi with RetroPie, so that she can play Road Warrior, an old Nintendo game that I used to play when I was a kid.

Swapping those SD cards is annoying, though. And having lost a lot of work on SD cards before has made me not trust the Pi as much when I need something to be a bit more permanent.

So I read that it is easy to get the Raspberry Pi booting off the network, and I decided it was time to try it.

I followed these two guides:

It worked fine, with two caveats:

My Pi would not get the TFTP address from dnsmasq
NSF 4 didn’t work for me (and it still doesn’t)

I use TrueNAS at home, and I have a Unifi Security Gateway, which allow me to configure some DHCP options. Doing that made my life a bit easier, actually, in that I only had to configure the NFS share and the TFTP service, and things worked.

So I’ll try to sum up how I got this working, without dnsmasq and using my TrueNAS.

Goal

My goal is to boot by Raspberry Pi 400 off my network. I want to boot RetroPie, since the obvious advantage is that I’d have even more space to roms if I need to. And this way, if I connect another Raspberry Pi to my TV, I could boot off the same share (if you do that, notice that you shouldn’t use both systems simultaneously)

List of used equipment/software

Raspberry Pi 400 (should work with a Raspberry Pi 4)
An SD card with RetroPie installed on it
FreeNAS/TrueNAS (for the NFS and TFTP services - you can use other servers, of course)
Unifi Security Gateway (should work with other DHCP servers - YMMV).

Creating the shares at the FreeNAS

First, we need to create the directories where we are going to put the files the Pi will boot off:

Create a new dataset on your pool. Give it a name (let’s say, pi). Accept the default options. So if your pool is called mypool, you will end up with a folder called /mnt/mypool/pi

On the shell, do this:

$ mkdir -p /mnt/mypool/pi/boot
$ chmod 777 /mnt/mypool/pi/boot

Note that this is assuming I will not boot other Pi’s with different OS’s. If you want to do that, it is probably best to create a directory with the serial number of the Pi you want to boot. See Step 6 here. The Pi first looks for its files on the remote computer on a folder with its serial number. If it doesn´t find one, it looks for the root folder. I am using this latter approach.

Now, we are going to configure the TFTP service, which is used by the Pi to load the OS:

On the Services menu of your FreeNAS/TrueNAS, enable TFTP, and edit it.
Choose the /mnt/mypool/pi/boot as the folder to be shared by the TFTP service.

Then we have to configure the NFS share.

On the Sharing menu, choose Unix shares (NFS), and click on Add
On Path, choose /mnt/mypool/pi
Click All dirs, and then Advanced options
On Maproot user, choose root, and on Maproot group, choose wheel.
Choose the network and/or hosts that will be allowed to mount your share

Note that security-wise these might not be the best permissions. Proceed with caution!

There! Now your FreeNAS is configured to serve your Pi! It needs the files, though…

Transferring the files from the Pi

The tutorials I referred to at the beginning of this article are based on an installation of a fresh image of an OS (Raspbian). I would rather copy an existing OS that I have installed on an SD card.

To do that, I had to copy the contents of the two partitions of the SD card - the root and the boot partitions.

I use a Mac, so the root partition is not automatically readable on macOS. But if you are using an Intel-based mac, you can do this:

$ brew install cask osxfuse
$ brew install ext4fuse

This way, you can mount the root partition of the SD card after inserting it on your SD card reader:

disklist list # check what's the name of the linux partition of the SD card - for example, disk2s2
sudo ext4fuse /dev/disk2s2 /tmp/raspberry

Caveat: This method has the problem that the directories are mounted as read-only. I changed a few (such as /home) recursively so that they could be writeable again after copying. If you have access to a Linux machine, you’re better off just copying the files from there instead.

If all works fine, you now have the files you need. Let’s copy the boot folder to the FreeeNAS:

$ scp -r /Volumes/boot/* myuser@myfreenasaddress:/mnt/mypool/pi/boot/.

However, due to the incompatibility of some filenames between Mac and Linux, we have to do an extra step to copy the root partition:

brew install gnu-tar
sudo gtar czf rasp.tgz /tmp/raspberry
scp rasp.tgz myuser@myfreenasaddress:/mnt/mypool/.

Now, on your FreeNAS:

cd /mnt/mypool
tar -xzf rasp.tgz
mv raspberry/* pi/* 

Adjustments on the configurations for booting

Good? Good. We just need to substitute two files on your FreeNAS:

cd /mnt/mypool/pi/boot
rm start4.elf
rm fixup4.dat
wget https://github.com/Hexxeh/rpi-firmware/raw/stable/start4.elf 
wget https://github.com/Hexxeh/rpi-firmware/raw/stable/fixup4.dat 

Somehow, the /home/pi folder got the wrong permissions set when copying the files. I believe it is because, on a Mac, ext4 partitions are read-only (when mounting with ext4fuse), so that’s probably why it happened. So let’s fix this:

$ chmod -R 750 /mnt/mypool/pi/home/pi

Now, we going to configure the pi to mount your nfs share. Edit the cmdline.txt:

$ nano /mnt/mypool/pi/boot/cmdline.txt

Erase the existing configuration andcopy this one:

console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.1.110:/mnt/mypool/pi,vers=3 rw ip=dhcp rootwait elevator=deadline

There are few other configurations from the RetroPie install on this cmdline.txt file, such as consoleblank=0. This one, for example, is safe to keep, though I also removed it so I can see if there are errors while booting.

And finally, edit the /mnt/mypool/pi/etc/fstab and remove the two lines you see there with UUID mountings.

There! You got the FreeNAS all configured to serve your files. Now, let’s get your Pi to boot off the network.

Configuring the Pi firmware

In the tutorials I mentioned, you need to download a newer firmware to your Raspberry Pi 4 to get things working, but most Pi’s now have a newer firmware and thus are able to boot off the network. You need to configure it, though. Boot your Pi with an existing Raspbian installed, and type

$ rpi-eeprom-config --edit

Add this option:

BOOT_ORDER=0x21

This will make the Pi to attempt booting off the SD card and, if it isn’t present, it will then try to boot off the ethernet. You have other boot options. For more information, check the documentation of the bootloader.

Now, we can configure something else: if you don’t want to install dnsmasq, or don’t want to configure your DHCP server to send the FreeNAS (TFTP) address to your pi on boot, you can add this configuration as well:

TFTP_IP=192.168.1.110

Change the above IP address to the one corresponding to your FreeNAS. If you choose to add this, you are all set, actually. Just save the configuration, reboot, then reboot again, this time without your SD card.

If you’d rather have your DHCP server to send the TFTP address to your Pi, read on.

Configuring your Unifi DHCP to send TFTP information

Open your Unifi Controller on your browser, and go to the Networks session on the Settings menu. Choose the network where your Pi will be configured (for example LAN), and click on “Advanced DHCP options”. There, enter the address of your FreeNAS on the DHCP TFTP Server field:

Save it, and bang, you are done. Now you will be able to boot your Pi off your RetroPie! I have used a bit now, and it is working as it should. I didn’t manage to upgrade it, though, probably due some permission issues, so I boot off the SD card again, did a sudo apt update and a sudo apt dist-upgrade and redid the copying of the files to the NAS. The best is if you install RetroPie from the scratch, using the instructions on the tutorials I mentioned (they also apply to the RetroPie, as it is Raspbian-based).

Conclusion

I really enjoyed this. I didn’t seem to get a much speedier boot, since the Raspberry 4 already reads the SD card at a higher speed. But the flexibility of dual booting, and also of being able have much more storage for my RetroPie, made this really worth it.

I really hope that someone comes up with a tutorial on how to get a PXE-menu working. Probably using the UEFI firmware would help here, but I am not sure my OS’es would boot when UEFI is loaded, and it seems it also needs to boot the UEFI from an SD card, which is something I really want to avoid.

Fiber at home - an adventure

Sat, 06 Mar 2021 15:20:23 +0100

A little background

I am lucky to be inspired by very smart people at work, all the time.

Almost three years ago, I wanted to buy a NAS (Network Attached Storage), as having hard isks connected to a router’s USB port wasn’t cutting it anymore. I had my eyes on a QNAP when a colleague came in with an older HP desktop and said “hey, try FreeNAS on this one, just to make sure a NAS is your thing”. Little did I know that it was the start of a lot of work, learning and joy (some of it described on my post about building my homelab machine).

It turned out that the HP desktop was too loud, not something I could keep on our living room. So I was ~~ordered~~ asked politely to move it to the basement. Luckily my wifi still could reach the basement, but a NAS on WiFi is not a thing, really. So I was not able to play with the cool stuff like Nextcloud when the speeds were that bad.

I then applied to the board of our housing association to be able to pull a cable from my flat to the basement. I got an approval, but under the condition that I had to find an existing duct to pull the cable. I wasn’t allowed to drill or otherwise make a new duct. “Piece of cake”, I thought, not knowing it was going to take me almost 2 years to get it done.

Three electricians came here, plus a couple of smart friends. Nobody saw anything usable. We have old ducts used for antennas, but they go upwards, and while we also have a storage room on the loft, we have no power there to plug stuff. I thought I could use the same duct used for the cable TV, but it turns out that one also goes up to the loft.

I was frustrated, as I really wanted to use my NAS. I focused then on building a silent machine that I could have on the living room, and that I managed to do, as described on the link above.

But with the pandemics came home office, and I have to work on the living room. That low humming of the hard disks (which is the only noise coming out from that machine) was really annoying me. I had to fix it.

I then asked permission to the board to get power on the loft, since all the ducts seemed to be going there. They gave me the permission - amazing board! - but recommended me to wait until august (this was like may 2020) since a company was going to upgrade the lights and electricity of the whole building, so it would make sense to get this done by the same company. Deal.

Things got postponed, and the company started its work on our building only in December. But, as they say, good things come to those who wait: I decided to use fiber optics instead of an ethernet (cat6/7) cable, as my motherboard have SFP+ ports, and I thought it was more future proof. It turns out that the company had an employee who worked in telecom for decades and could do the job for me.

The odyssey to get fiber spliced

The electrician came to see the flat, and to check the loft to see how to get power there. He thought it would be hard to use the duct used for tv cables, as it seemed to be tight there. And then he asked, “why don’t you use the cellar?”. I explained to him that I didn’t find any existing duct I could use to pass the cable through to the cellar.

While we took the stairs down, he immediatelly looked about my entry door and asked, “what is this?”. That was a little box between the doors of the two flats. The man worked in telecom for decades, as I said, so he knows his stuff. He took some stairs, opened the little box, and bingo! He found a duct to the cellar, which was not in use and had just old telephone cables there.

This is the old phone duct:

We agreed then we would do the job in January 2021, as it seemed to be possible to pass the fiber cable through that duct.

But then my luck went way: his company couldn’t find the appropriate cable (I wanted OM4, as I wanted to use multi mode fiber, which I kinda regret now, but I already had the multi mode transceivers). I decided to buy the cable myself.

If you try to buy anything related to fiber optics in Norway as a private customer (ie. not as a company), you are basically out of luck. But finally I found a company that accepted to sell me the cable I wanted, and I had to take a 40 min ride on a bus to go there, walk on slippery roads and get the cable home. I called the guy to ask when we could start pulling the cable, but it turns out he was on a sick leave and wasn’t going to work before a few weeks. I was on my own.

I asked the company where I bought the cable if they could recommended me of anyone that could do the job for me, and they gave me a name. I was skeptical, because EVERY single company I called either didn’t reply or wouldn’t take my business. But this guy agreed to come and see my building.

Then we got into another problem: the duct ends up on a box on the basement that nobody had a key for.

The guy, as the one previously, said he might have a key for it (it is an old box that was used back in the days by Telenor - the Norwegian telecom incumbent). Suddenly the guy disappeared! He wouldn’t reply my sms’s, and I was on my own again.

I had no idea how to open this box:

Another colleague recommended an electrician, but this one wouldn’t be available before two months. He had a colleague that would call me instead, but never did.

Fortunately, my colleagues are not just a source of inspiration, they helped me a lot. The same one who encouraged me to start with FreeNAS offered to see if we could pull the cable ourselves. That way, I would save a lot with the costs.

It was tough: very tight ducts, lots of old wires. Still, we were not sure we could make it through the box. I bought a fish tape and an endoscopic camera to see if we could avoid opening the box, but that was a no go.

Now I had to ask the board permission to ask a locksmith to open the box and change the key so we could open it. I was really relieved when I got the permission to do it. I called a locksmith, and after a few days he came and opened the thing.

Now my friend came home to help me, and we tried to pull the cable through the duct, and for a few minutes we thought it wasn´t going to go through, but after a few attempts, we finally reached the cellar!

We then pulled the fiber cable with the help of the fish tape, and it reached my storage room! Perfect! Now I need to get this fiber spliced!

Oh, the joy to see that cable come through…

The thing is that I bought the cable terminated on one side, but it had to be terminated on the other side, as it would never pass the ducts with the connectors.

My saga with companies specialized in fiber optics restarted: no company would splice the fibers for me, and those who had splicing machines for rent would not rent them to me (they said they only rent to other companies, not to private customers). I was getting desperate.

I decided then to press the “F..it” button and buy a cheaper (which is still expensive) fiber splicing machine. Saw some on Ali Express and Banggood, and decided to get the same model from Amazon, as the delivery was faster and because, well, I trust Amazon a bit more than individual sellers on Ali Express.

My plan was to buy the machine, splice my cable, make sure it works, and then sell the machine. I’ve put it on the shopping cart on Amazon, went to the kitchen to take the bread I was baking out of the oven, and when I was coming back to finalize the shopping, I got a call from the electrician who was on a sick leave! He was back, happy that I managed to pass the cable, and he could come in two days to splice the fiber! I can’t tell you what a relief that was.

He came last Friday, spliced the fibers (I got two pairs), and had everything beautifully installed down the basement. He asked me to test, and to tell him if anything was wrong.

When he left, I tested, it didn’t work. Tested again, no signal. On the third time, after I inverted the connects, it worked! I got now one (possibly two, I haven’t tested the other fiber pair yet) 10Gbits/sec connection to my basement!

I got my big NAS box out of our living room, connected to the switch in the basement, and came running upstairs to see if everything was ok. Ah, the joy of a quiet living room…

Aftermath

I am pretty happy with my setup. It was totally worth it. Oslo is a very expensive city, so space use has to be optimized. Being able to use my storage room to house some equipment is perfect: cool temperatures, no problems with noise, and room to experiment a bit with other equipment if I want to do so.

However, the bitter taste is still here: it shouldn’t have to be this hard. It is appalling to think how hard it is to get a bit of a slightly more specialized equipment when you are not a company. I could have ordered a lot from overseas, which I did in fact - fs.com has an amazing service and sold me some of the patch cables I needed. But it is shocking that if a user wants to install fiber for using on an internal LAN, it might be very hard to buy the components or the get someone to do the fiber service. A part of me wishes that I should have bought that machine, so that I won’t have to rely on the good will of companies to help me if I need to service the cable, replace it, etc.

Configuring a Linux image for Horizon instant clones, including Active Directory and NFSv4

Tue, 24 Nov 2020 17:20:23 +0100

A pool of instant clones of VDI (Virtual Desktop Interface) is the one where an image of an OS is cloned so that several users can login to a similar computer set-up. Normally, these clones are destroyed and recreated, so it is common to setup an strategy for persistent storage.

VMware has provided some documentation on how to configure Linux machines for those pools of instant clones, and arguably the most challenging part is the integration with Active Directory. Most of the documentation is linked from a document called Integrating Linux with Active Directory. However, I’ve found that the documentation provided by VMWare can be a bit outdated or incomplete.

One example: On this document, VMware doesn’t mention that the clones need to rejoin the domain when created, though on another document describing the procedure for RHEL this step is indeed mentioned.

Another problem is that VMware skips the simplest of the situations, which is that AD can simply be joined by configurind SSSD correctly using ad as id_provider, auth_provider and access_provider, and not necessarily having to use Winbind, Samba, OpenLDAP or LDAP authentication to achieve this. In other words, you can still join AD without needing to use Samba, Winbind or LDAP, so there is a way to join AD in a more straightforward way.

This document attempts to supplement the information found elsewhere on how to properly configure a Linux OS image for deployment of an instant clones-pool. I used RHEL 8 here, but most of the things I mention can be used on other distros, though RHEL 8 is supported officially by Horizon 2006.

Summary of the steps you need and what isn’t covered here

I’m going to explain the following steps, which is what you need to configure your Linux image in order to get instant clones:

Join the base (called golden image on VMWare documentation) to Active Directory
Optionally configure pam_mount so that your users get home directories mounted from an NFSv4 server
Make a decision on how you are going to save your credentials on the gold image so that the cloned images can (re)join the domain
Write a script to orchestrate the rejoining
Change the Horizon View agent configuration to run that script you created for rejoining
Shutdown the image, create a snapshot if it doesn’t have one.

With these steps, you can deploy your image.

Things I don’t mention here but I assume you have done them already:

Create a user on AD for the purpose of joining other machines to AD
Create the configuration for kerberos on /etc/krb5.conf
Install the Horizon View Agent for Linux
DNS is working fine, the hosts resolve to the domain you’re using, etc.

Ready? Let’s do it!

Setting up the image

Configuring your `/etc/sssd/sssd.conf`

As I said, VMware does not mention using AD as a source of authentication. Configure this by editing your /etc/sssd/sssd.conf. Here is an example of a working configuration:

[sssd]
config_file_version = 2
domains = mydomain.com
services = nss, pam, pac, sudo

[nss]
allowed_shells = *
default_shell = /bin/bash
shell_fallback = /bin/bash
homedir_substring = /home
override_homedir = %H/%u

filter_groups = root,apache,mysql,postgres,store,palantir,palconf,mailman
filter_users = root,apache,mysql,postgres,store,palantir,mailman,ghost,priss

[domain/EXAMPLE.COM]
id_provider = ad
auth_provider = ad
access_provider = ad
autofs_provider = ad
chpass_provider = ad
ldap_id_mapping = false # You might want to set this to true, but if you have uidNumber and gidNumber on your users' records, it works well with false.
enumerate = false
dns_discovery_domain = example.com

krb5_realm = EXAMPLE.COM
krb5_renewable_lifetime = 14d
krb5_renew_interval = 3600
krb5_ccname_template = FILE:%d/krb5cc_%U
ad_gpo_map_interactive = +gdm-vmwcred

Not all settings above are mandatory, but those are the ones that worked for us.

Join the domain

You can join the domain on a RHEL 8 install by using the realm join, but it configures a lot of the things which we already configured by hand on the sssd file. If you perform a realm leave, it ends up erasing your configuration, so I decided to use adcli.

But before joining the domain, it’s important to decide if, when rejoining the domain on the clones, you are going to save the credentials or if you are going to use the password. Both have its advantages and disadvantages.

Using the password can be interesting if you don’t plan to build your images too often. However, it is less secure, especially if the user you use to join AD has other capabilities.

Using a password to join the domain is simple:

$ adcli join -U username@EXAMPLE.COM -D EXAMPLE.COM --service-name=nfs

Note: the --service-name=nfs was used in case you are using an nfs service.

Enter your password. Then if all goes well, you will have joined the domain. Test if you got the right tickets:

$ klist -ke

If, however, you prefer to cache your credentials on the image so that it can be reused by the clones for joining the domain, follow the instructions below. Using cache means that you need to rebuild the image so that clones can be created before these cached credentials get expired.

$ kinit username@EXAMPLE.COM -f -A -r 30d -l 30d -c /root/mycachedcredentials
$ adcli join --login-ccache=/root/mycachedcredentials -D EXAMPLE.COM --service-name=nfs

There, no need of typing a password, as your credentials were read from the mycachedcredentials file. You can save it whenever you want.

If all goes well and you joined the domain, then let’s make the final configurations.

Configuration for the Horizon View Agent

Ok, as I said before, the clones need to rejoin the domain, right? This is because they will have their own hostname, other ip addresses, etc., so it needs to join AD again. We need to orchestrate this. Luckly, VMWare helps us here with some settings.

What you do want to do now is to create a script that will configure your clones with some settings and join the domain.

Here’s a sample script that does the trick:

#! /bin/bash
# (Re)joins machine to AD after hostname is changed when instant clones are created.
# Must be called by /etc/vmware/viewagent-custom.conf

LOG=/tmp/ad-join.log # Because logging never killed anyone...
fqdn="$(hostname).example.com"
hostname="$(hostname)"
echo "This instant clone will joining ad..." >> $LOG
echo "Host: $(hostname)" >> $LOG

# Set new hostnames
hostname "$fqdn"
cat <<EOF > /etc/hosts
127.0.0.1 $hostname
::1   $hostname
EOF
echo "$fqdn" > /etc/host

# Stop SSSD so that the previous keytab is released from the cache
service sssd stop
sss_cache -E
rm -rf /var/lib/sss/mc/*
rm -rf /var/lib/sss/db/*
rm -rf /var/lib/sss/pubconf/*

# (Re)join AD
rm -rf /etc/krb5.keytab
adcli join --login-ccache=/root/mycachedcredentials -D EXAMPLE.COM --service-name=nfs
test $? -eq 0 || echo "Joining failed" >> $LOG
sleep 5
rm -rf /root/mycachedcredentials # Why leave your credentials on that clone? 

# Restart SSSD so we can authenticate again
service sssd start

Save this as, for example, /usr/local/bin/ad-join.sh.

If you want to use a password instead of the cached credentials, change tthe adcli line above to:

echo "myfancypassword" | adcli join --stdin-password -join -U username@EXAMPLE.COM -D EXAMPLE.COM --service-name=nfs

There is one detail that you might have seen here: we’re stopping sssd and removing all its cache. This is important because the older keytab is still referred to by sssd. You get plenty of errors like this one if you don’t stop sssd and refresh the cache prior rejoining the domain:

Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Preauthentication failed. Unable to create GSSAPI-encrypted LDAP connection.

This error was driving me crazy, and if you google it, most of the information will tell you that you need to remove the /etc/krb5.keytab, delete the account on AD, etc., but actually most of these things won’t be necessary (Note that I do remove the old keytab). Stopping sssd and removing its cache prior rejoining the domain did the trick to fix this. Credits to this forum post.

Now, one last step. Open /etc/vmware/viewagent-custom.conf and edit the following lines:

...
RunOnceScript=/usr/local/bin/ad-join.sh # or whatever the path/name you chose for your script
RunOnceScriptTimeout=120 # Adjust here if you believe that your rejoins take more than 2 minutes.
...

That’s it! Now your image is fully configured for joining the domain. Shut it down, and create a snapshot with your modifications on vCenter, and schedule it to boot from your Horizon connection server.

But if you want to mount NFS shares as home directories, do the following step before shutting down your machine:

Setting up `pam_mount` for mounting home directories

There are a few ways to automount home directories, such as autofs. I used pam_mount, but might change it in the future for the flexibility of autofs.

Using NFSv4 with kerberos is a very good solution to mount those directories. If you got a kerberized NFSv4 server, why not?

With pam_mount, this was my configuration at /etc/security/pam_mount.conf.xml:

 <volume
 fstype="nfs"
 server="serveraddress"
 uid="10000-100000" 
 path="/homedir/%(USER)"
 mountpoint="/home/%(USER)"
 options="vers=4,sec=krb5"
 noroot="0"
 />

Note the id="10000-10000" - remove it or adjust it accordingly. It basically means that we restrict mounting attempt to those uid’s.

Now a last configuration: . On the file /etc/idmapd.conf, enter your domain:

Domain = example.com

Lastly, add the following line to the end of your /etc/pam.d/password-auth file in order to use pam_mmount:

session     optional                                     pam_mount.so disable_pam_password disable_interactive

Things that might go wrong with pam_mount:

SELinux is a $@#!$. Check your logs. If you use the logout configuration of pam_mount, it might require some adjustments of SELinux - you will which ones on your logs. I had to add two exceptions for SELinux when using pam_mount, one of them related to ofl.
Check your DNS. Do you get reverse domain lookups working, for example?

Final remarks

I hope this can help supplementing the information found on VMWare docs on how to integrate Linux to Active Directory with the purpose of offering pools of instant clones.

Any remarks, comments or tips? Feel free to comment!

Francis Augusto Medeiros-Logeay

Platform Single Sign-on DIY

What?

Disclaimers

What is Platform SSO?

Implementing Platform SSO from the perspective of an IdP

Requirements for IdPs and how we did it

Requirements for your PSSO Extension

How we did it on the IdP

The nonce endpoint:

The device registration endpoint

The user registration endpoint

The token endpoint

When does the PSSO extension call these endpoints?

The authentication itself

A few things that don’t work well yet

Conclusion

Acknowledgments

How to use Keycloak as an IdP for Entra ID via ADFS

What?

Disclaimers

Why?

Introduction

Configuring ADFS to use Keycloak as an IdP

Claims, claims and more claims

2FA or not 2FA? That’s the question

Conclusion

Appendix

E-mail authentication, quo vadis? (Or where is OIDC/Oauth2 for everyone?)

The complexities of running a Fediverse instance on clusters

Disclaimer

Introduction

Cluster deployability

Documentation

Community and help for sysadmins

Conclusion

Installing Matrix on Kubernetes - what they don't tell you

Disclaimer

Why are you writing this?

The workers

Load balancing

Other things

Conclusion

Babb.no - The Fediverse and open social networks

Introduction

Nerdy talk

Conclusion

Running a Mastodon instance on kubernetes

Introduction

The challenge

How

Remained things to do:

Booting a Raspberry Pi 400 from FreeNAS with Unifi

Introduction

Goal

List of used equipment/software

Creating the shares at the FreeNAS

Transferring the files from the Pi

Adjustments on the configurations for booting

Configuring the Pi firmware

Configuring your Unifi DHCP to send TFTP information

Conclusion

Fiber at home - an adventure

A little background

The odyssey to get fiber spliced

Aftermath

Configuring a Linux image for Horizon instant clones, including Active Directory and NFSv4

Summary of the steps you need and what isn’t covered here

Setting up the image

Configuring your /etc/sssd/sssd.conf

Join the domain

Configuration for the Horizon View Agent

Setting up pam_mount for mounting home directories

Final remarks

The `nonce` endpoint:

Configuring your `/etc/sssd/sssd.conf`

Setting up `pam_mount` for mounting home directories