Introduction to Internal Developer Platform

My career started quite late; I began working only after coming to Japan, having previously been an entrepreneur. I feel embarrassed to admit that my business never scaled significantly, with a maximum of just over ten people, so I lack experience in managing large-scale software engineering teams. This knowledge has always been lacking. After switching to a new company, I finally understood what an Internal Developer Platform (IDP) is, and I want to document it here.

In my view, the purpose of an IDP is to enable continuous delivery of reliable software in the context of large-scale collaboration among software engineering teams (at least hundreds of people). I checked internaldeveloperplatform.org, and while there are some out of the box frameworks, generally speaking, each company has its own considerations based on its business when choosing technologies, so almost all IDPs are built by the company's infrastructure team. Therefore, it is indeed difficult to understand what IDP is, its benefits, and why to adopt it without working at a company that has one.

According to internaldeveloperplatform.org, IDP has five Core Components: Application Configuration Management, Infrastructure Orchestration, Environment Management, Deployment Management, and Role-Based Access Control. This explanation is too dry, so I will illustrate it with specific examples. For ease of understanding and to anonymize information, I will simplify certain aspects.

A common implementation of IDP is to converge application code, infrastructure configuration, deployment configuration, and permission governance into a unified GitOps process. This article uses monorepo + Bazel + GitHub + Kubernetes + Argo CD as an example.

Monorepo

The first thing that catches the eye in IDP is a massive monorepo where all source code for all projects from all teams is stored. Due to its size, it must be managed using Bazel. This monorepo contains not only software source code but also Terraform configurations (i.e., Infrastructure as Code), Kubernetes configurations, and various GitHub configurations. Bazel is a build tool developed by Google, specifically designed for extremely large monorepos, supporting all languages, frameworks, and projects.

Software Code Updates

Alice is a programmer on team0, and her project is located in projects/foo within the monorepo. She has completed a feature and pushed it to the foo/feature-x branch, submitting a PR. At this point, to ensure the quality of the delivered code, the GitHub Actions developed by the infrastructure team are triggered immediately, automatically running Bazel scripts to perform linting, CI, compilation checks, etc. If any of these tasks fail, the code cannot be merged, forcing Alice to write better code.

Even if all GitHub Actions pass, Alice cannot merge the code on her own. Companies using IDP are large, and they have their own processes. When a PR is initiated, GitHub automatically requires code reviews from the GitHub organization teams, specifically from people in team0. How does GitHub know that team0 needs to review foo? Because it is configured in the CODEOWNERS file at the root of the monorepo.

After receiving the email notification from GitHub, Bob clicks approve and comments 'lgtm', at which point the PR can be merged. Anyone in team0 can click the merge button.

After merging into the main branch, the GitHub app developed by the infrastructure team is triggered immediately to check if a new Docker/OCI image needs to be packaged. If so, it starts packaging through Bazel. Once packaging is complete, it automatically pushes to the internal Docker registry (such as GCP's Artifact Registry) and automatically initiates two PRs to merge into main, modifying the image tag in the Kubernetes deployment configuration in infra/k8s/foo/xxx.yaml. Why two? Because one modifies the YAML for the dev/staging environment, and the other modifies the YAML for the prod environment. Each PR includes the commit on which this image build is based, facilitating traceability.

According to CODEOWNERS, GitHub will send emails to team0. Alice or Bob can click approve and execute the merge after seeing the email. They can choose to merge only the PR that modifies the dev/staging environment, merge the one that modifies the prod environment, or even merge nothing at all. This is entirely their team's decision, and the infrastructure team does not intervene.

If the PR modifying the image tag is merged into main, Argo CD will immediately sync and attempt to complete the deployment.

Dependency Updates

The infrastructure team learns that a certain npm package has been compromised and responds immediately. Thanks to the unified management of all dependencies in the monorepo, they only need to update the package.json, push it to infra/update-x-deps, and merge it into main, following the same process as described above. After the main branch is updated, since foo depends on this npm package, the previously mentioned GitHub app will automatically repackage foo's image and send an email to team0 members, Alice and Bob, who are responsible for clicking merge to trigger the Argo CD deployment. Not only foo, but all other projects that depend on this npm package will also automatically execute the same process.

If the infrastructure team decides to upgrade the base dependencies of a certain language/framework's OCI image, such as the Debian version, they only need to update the Bazel configuration and execute the above process again. After merging into main, Alice and Bob will receive notifications to decide whether to deploy.

In this way, the maintenance of base dependencies can be entirely entrusted to the security operations team, allowing the application development team to focus solely on business implementation.

New Software Launch

Carol from team1 needs to create a new project called bar and requires a PostgreSQL database. Carol needs to:

• Create a new project directory under the projects directory in the monorepo and maintain the BUILD file needed by Bazel.
• Create database-related files in the Terraform-related directory of the monorepo.
• Create bar-related deployment in the infra/k8s directory of the monorepo and input the environment variables. If the environment is a secret, it needs to be managed through HashiCorp Vault.
• Create domain-related configurations in the Terraform and infra/k8s directories of the monorepo.

Once everything is prepared, Carol initiates a PR to merge into main. These PRs may not all be handled by Carol; people from the infrastructure team may assist with the Terraform parts. In an ideal scenario, a web UI (like Backstage) could encapsulate the above GitOps, making it as simple as using SaaS software for internal developers.

Similar to the above, once the PR is initiated, GitHub Actions will immediately execute CI, Bazel scripts, lint checks, etc. Since Terraform has been modified, CI will also check tf-plan. If any of these CIs fail, the merge cannot proceed. According to CODEOWNERS, changes related to Terraform require code review from the infrastructure team to ensure clear maintenance responsibilities.

Software Access Restrictions

All software from all teams is deployed in a single (dev/staging/prod) Kubernetes cluster, with different projects simply having different namespaces, allowing for potential mutual access. However, for security reasons, such access must be prohibited by default. The Kubernetes portion in the monorepo can define a set of Istio rules, allowing each team to override them in their own Kubernetes definition directories. CODEOWNERS can be configured so that similar changes require code review from the infrastructure team when merging into main.

Interface Reuse

Some software is intended for internal use, such as internal SDKs. Once these SDKs are deployed to Kubernetes, they have their own internal and external access addresses. The aforementioned Backstage can not only simplify the software launch process but also provide clear internal documentation and architectural overviews for team members, effectively avoiding duplicate development.

Among the most important aspects is observability; dev/prod clusters often have various collectors that forward data to platforms like Grafana Cloud.

Another important aspect may be internal SSO; the company might deploy an OIDC provider similar to Keycloak, allowing other services to integrate without needing to write any registration or login, unifying access to internal interfaces.

Revisiting the Five Core Components

Combining the above specific examples, let's see how IDP integrates various industry tools to solve the following problems:

Application Configuration Management

This is mainly addressed through Kubernetes environment configurations. The secret parts can be managed through HashiCorp Vault, but it is quite expensive. I found alternatives like dotenvx and envsecrets. Absolutely no keys should be directly introduced into the project source code.

Infrastructure Orchestration

This is mainly solved through Terraform and Kubernetes resource requests/limits, node affinity, and even KEDA.

Environment Management

This is addressed by preparing different dev/staging/prod Kubernetes clusters and binding them to different parts of the monorepo through Argo CD.

Deployment Management

This is managed through Argo CD.

Role-Based Access Control

Code change governance is mainly implemented through GitHub Teams, branch protection, rulesets, CODEOWNERS, and required status checks. Runtime access control is achieved through Kubernetes RBAC, namespace isolation, service accounts, network policies, or Istio AuthorizationPolicy mechanisms.

Conclusion

This article is just a starting point. Currently, I am not part of the infrastructure team; I am merely a user of IDP, making some inferences from an external perspective based on my experience. To anonymize information, I have made many simplifications and modifications. For example, the IDP at my current workplace does not have a web UI, but I believe there should be one. It should be quick to wrap GitOps in a web shell for coding agents. IDP is closely related to business and does not have universally applicable standards. Different companies' organizational structures and even office politics can influence the final form of IDP.

In my own entrepreneurial projects, since I am currently a solopreneur and only responsible for myself, I have greatly simplified GitOps. I do not require any reviews; reviewing myself is not meaningful since I am only accountable to myself. I just need to push, which immediately triggers GitHub Actions to run CI. If CI passes, it packages, and once packaging is complete, it automatically updates the image tag, and then Argo CD automatically deploys. Additionally, I do not need to use a heavyweight tool like Bazel; npm packages are sufficient, and Rust and Python projects can also utilize npm scripts.