Introduction to Internal Developer Platform

My career started late; I began working only after coming to Japan, having previously been an entrepreneur. I feel ashamed to say that my business never scaled significantly, with a maximum of just over ten people, so I lack experience in managing large-scale software engineering teams. This knowledge has always been lacking. After switching to a new company, I finally understood what an Internal Developer Platform (IDP) is, and I want to document it here.

In my view, the purpose of an IDP is to enable continuous delivery of reliable software in the context of large-scale collaboration among software engineering teams (at least hundreds of people). I checked internaldeveloperplatform.org, and although there are some out of the box frameworks, generally speaking, each company has its own considerations based on its business when choosing technologies, so almost all IDPs are built by the company's infrastructure team. Therefore, it is indeed difficult to understand what an IDP is, what its benefits are, and why to adopt one without working at a company that has an IDP.

According to internaldeveloperplatform.org, IDP has five Core Components: Application Configuration Management, Infrastructure Orchestration, Environment Management, Deployment Management, and Role-Based Access Control. This explanation is too dry, so I will illustrate it with specific examples. For clarity and to anonymize information, I will simplify certain aspects.

A common implementation of an IDP is to converge application code, infrastructure configuration, deployment configuration, and permission governance into a unified GitOps process. This article uses monorepo + Bazel + GitHub + Kubernetes + Argo CD as an example.

Monorepo

The first thing that catches the eye in an IDP is a massive monorepo, where all the source code for all projects from all teams is stored. Due to its size, it must be managed using Bazel. This monorepo contains not only software source code but also Terraform configurations (i.e., Infrastructure as Code), Kubernetes configurations, and various GitHub configurations. Bazel is a build tool developed by Google, designed specifically for large-scale monorepos, supporting all languages, frameworks, and projects.

Software Code Updates

Alice is a programmer on team0, and her project is located in the monorepo under projects/foo. She has completed a feature and pushed it to the foo/feature-x branch, submitting a pull request (PR). At this point, to ensure the quality of the delivered code, the GitHub Actions developed by the infrastructure team are triggered immediately, automatically running Bazel scripts to perform linting, continuous integration (CI), compilation checks, and vulnerability detection for new dependencies. If any of these tasks fail, the code cannot be merged, forcing Alice to write better code.

Even if all GitHub Actions pass, Alice cannot merge the code herself. Companies using IDPs are typically large, and they have their own processes. When a PR is initiated, GitHub automatically requires code reviews from members of the GitHub organization teams, specifically from team0. How does GitHub know that team0 needs to review foo? Because it is configured in the CODEOWNERS file at the root of the monorepo.

After Bob receives the email notification from GitHub, he clicks approve and leaves a comment saying "lgtm" (looks good to me), at which point the PR can be merged. Any member of team0 can click the merge button.

Once merged into the main branch, the GitHub app developed by the infrastructure team is triggered immediately to check if a new Docker/OCI image needs to be packaged. If so, it begins packaging via Bazel. After packaging is complete, it is automatically pushed to the internal Docker registry (such as GCP's Artifact Registry), and two PRs are automatically initiated to merge into main, modifying the infra/k8s/foo/xxx.yaml file in the monorepo to update the Kubernetes deployment's image tag. Why two? Because one modifies the YAML for the dev/staging environment, and the other modifies the YAML for the prod environment. Each PR includes the commit on which this image build is based, facilitating traceability.

According to the CODEOWNERS, GitHub will send an email to team0. Alice or Bob can click approve and execute the merge after seeing the email. They can choose to merge only the PR that modifies the dev/staging environment or the one that modifies the prod environment, or even merge nothing at all. This is entirely up to their team, and the infrastructure team does not intervene.

If the PR modifying the image tag is merged into main, Argo CD will immediately synchronize and attempt to complete the deployment.

Dependency Updates

The infrastructure team learns that a certain npm package has been compromised and responds immediately. Thanks to the monorepo's unified management of all dependencies, they only need to update the package.json, push it to infra/update-x-deps, and merge it into main. The process is the same as described above. After the main branch is updated, since project foo depends on this npm package, the previously mentioned GitHub app will automatically rebuild the image for foo, push it to the image registry, and automatically initiate two PRs to merge into main, modifying the infra/k8s/foo/xxx.yaml file in the monorepo to update the Kubernetes deployment's image tag, and send emails to team0 members. Alice and Bob are responsible for clicking merge, triggering Argo CD deployment. Not only foo, but all other projects that depend on this npm package will also automatically execute the same process.

If the infrastructure team decides to upgrade the base dependencies of a certain language/framework's OCI image, such as the Debian version, they only need to update the Bazel configuration and execute the above process again. After merging into main, Alice and Bob will receive notifications about the PR updating the image tag and decide whether to deploy.

In this way, the maintenance of base dependencies can be entirely entrusted to the security operations team, allowing the application development team to focus solely on business implementation.

Software Launch

Carol from team1 needs to create a new project called bar, which requires a PostgreSQL database. Carol needs to:

• Create a new project directory under the projects directory in the monorepo and maintain the BUILD file required by Bazel.
• Create database-related files in the Terraform-related directory of the monorepo.
• Create deployment files related to bar in the infra/k8s directory of the monorepo and input the environment variables. If the environment variables are secrets, they need to be managed through HashiCorp Vault.
• Create domain-related configurations in the Terraform and infra/k8s directories of the monorepo.

Once everything is prepared, Carol initiates a PR to merge into main. These PRs may not all be handled by Carol; members of the infrastructure team may assist with the Terraform parts. Ideally, a web UI (like Backstage) could encapsulate the above GitOps processes, making it as simple as using SaaS software for internal developers.

Similar to the above, once the PR is initiated, GitHub Actions will immediately execute CI, Bazel scripts, lint checks, etc. Since Terraform has been modified, CI will also check tf-plan. If any of the CI checks fail, the merge cannot proceed. According to CODEOWNERS, changes related to Terraform require a code review from the infrastructure team to ensure clear maintenance responsibilities.

Software Access Restrictions

All teams' software is deployed in a single (dev/staging/prod) Kubernetes cluster, with different projects only differing by namespace. Therefore, theoretically, they can access each other. However, for security reasons, such access should be prohibited by default. The Kubernetes section in the monorepo can define a set of Istio rules, allowing each team to override them in their own Kubernetes definition directories. CODEOWNERS can be configured so that similar changes require a code review from the infrastructure team when merging into main.

Interface Reuse

Some software is intended for internal use, such as internal SDKs. Once these SDKs are deployed to Kubernetes, they have their own internal and external access addresses. The aforementioned Backstage can not only simplify the application launch process but also provide clear internal documentation and architecture overviews for team members, effectively avoiding redundant development.

The most important aspect of this is observability; internal dev/prod clusters often have various collectors that forward data to platforms like Grafana Cloud.

Another important aspect may be internal SSO; the company may deploy an OIDC provider similar to Keycloak, allowing other services to integrate without needing to implement registration and login, unifying access to internal interfaces.

Revisiting the Five Core Components

Combining the above specific examples, let's see how IDP integrates various industry tools to solve the following problems:

Application Configuration Management

This is primarily addressed through Kubernetes environment configurations. Secrets can be managed through HashiCorp Vault, but it is quite expensive. I found alternatives like dotenvx and envsecrets. It is absolutely essential not to directly include any secrets in the project source code.

Infrastructure Orchestration

This is mainly addressed through Terraform and Kubernetes resource requests/limits, node affinity, and even KEDA.

Environment Management

This is resolved by preparing different dev/staging/prod Kubernetes clusters and binding them to different parts of the monorepo through Argo CD.

Deployment Management

This is handled through Argo CD.

Role-Based Access Control

Code change governance is primarily achieved through GitHub Teams, branch protection, rulesets, CODEOWNERS, and required status checks. Runtime access control is implemented through Kubernetes RBAC, namespace isolation, service accounts, network policies, or Istio AuthorizationPolicy mechanisms.

Conclusion

This article is just a starting point. Currently, I am not part of the infrastructure team; I am merely a user of the IDP, making some inferences from an external perspective based on my experience. To anonymize information, I have made many simplifications and modifications. For instance, the IDP at my current workplace does not have a web UI, but I believe there should be one, as wrapping GitOps in a web shell for coding agents should be quick. IDP is closely related to business and does not have a one-size-fits-all standard. Different companies' organizational structures and even office politics will influence the final form of an IDP.

In my own entrepreneurial projects, since I am currently a solopreneur and only accountable to myself, I have greatly simplified GitOps. I do not require any reviews; reviewing myself is meaningless since I am only responsible for myself. I just need to push, which immediately triggers GitHub Actions to run CI. If CI passes, it packages the code, updates the image tag automatically, and then Argo CD deploys it automatically. Additionally, I do not need to use a heavyweight tool like Bazel; npm packages are sufficient, and Rust and Python projects can also utilize npm scripts.