Monitoring & troubleshooting

Every CR says whether it's in sync and, if not, why. This page explains how to read that, what to do about each reason, and how to watch the operator from your monitoring stack and from ArgoCD.

Check the statusCopy Anchor Link

For a quick overview, list the kinds with their READY column:

kubectl get fji                              # cluster-scoped kinds
kubectl get fjli,fjorg,fjteam,fjuser,fjukey,fjrepo,fjhook,fjdkey,fjoauth,fjtoken -A

When something reads False, look at the condition:

kubectl -n team-a describe fjrepo app

Every kind shares one status shape:

status:
  observedGeneration: 3
  forgejoId: "42"            # the Forgejo object this CR owns
  forgejoName: weebo/app     # the name Forgejo knows it by
  conditions:
    - type: Ready
      status: "False"
      reason: ReferenceNotReady
      message: ForgejoOrganization "weebo" is not ready yet (InstanceUnreachable)

The operator also records a Kubernetes Event for every error and every state change of a CR; kubectl describe and kubectl get events -n team-a show the history (Events lists them all).

Reason codesCopy Anchor Link

Each code, where it shows up, what it means and what to do about it is in Reason codes, generated from the operator's source so it always matches the installed version.

Disruptive changesCopy Anchor Link

Some changes can only be made by deleting and recreating the object in Forgejo, or they break whoever relies on its URL or credentials:

• transferring a repository to another owner;
• moving a team to another organization;
• changing an OAuth2 application's owner;
• changing an access token's scopes or repositoryRefs.

These are refused (DisruptiveUpdateRefused) until you opt in on the CR:

kubectl -n team-a annotate fjrepo app forgejo.weebo.io/allow-disruptive-update=true

DeletionCopy Anchor Link

spec.deletionPolicy decides what happens in Forgejo when you delete a CR:

Users are never purged, even with Delete. Secrets the operator generated are always deleted with their CR — only those it owns (forgejo.weebo.io/owner). A failed cleanup is retried and reported as a Warning Event (action Cleanup, reason of the failure — InstanceInUse, ForgejoApiError…); forgejo.weebo.io/force-orphan: "true" releases the CR without cleanup. A ForgejoInstance is only deleted once no CR is bound to it any more.

The operator never takes over an object that already exists in Forgejo. To bring existing objects under management, see High availability & sizing.

Metrics and alertsCopy Anchor Link

PrometheusCopy Anchor Link

Each replica serves Prometheus metrics in plain HTTP on port 9090 (GET /metrics), exposed by the weebo-forgejo-metrics Service. It's on by default (metrics.prometheus.enabled). With the prometheus-operator:

serviceMonitor:
  enabled: true
  labels:
    release: kube-prometheus-stack   # whatever your Prometheus selects on
prometheusRule:
  enabled: true

Every metric, its labels and example queries are in Metrics & health.

The metrics are recorded through OpenTelemetry and exported under the names listed there (counters end in _total, durations in _seconds). The chart's alerts match them with regular expressions, so they also catch the doubled suffixes (_total_total, _seconds_seconds) older releases exported.

Built-in alertsCopy Anchor Link

prometheusRule.enabled ships these (prometheusRule.defaultRules: false to drop them, additionalRules to add yours):

They filter on the scrape job weebo-forgejo-metrics in the release namespace, as set up by the ServiceMonitor. A failing reconcile counts as result error (or failure/failed).

OTLP traces and metricsCopy Anchor Link

Set OTEL_EXPORTER_OTLP_ENDPOINT through the chart's extraEnv to also export traces and metrics over OTLP/gRPC:

extraEnv:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector.observability:4317
  - name: RUST_LOG          # log verbosity, default info
    value: info

When something is wrong around the CRs rather than in them (webhook down, nothing reconciles), see Troubleshooting.

ArgoCDCopy Anchor Link

ArgoCD doesn't know how to tell whether these CRs are healthy. Without a health check, they show as Progressing forever. The chart ships one in the weebo-forgejo-argocd-health ConfigMap. Add it to ArgoCD's argocd-cm once per kind, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.forgejo.weebo.io_ForgejoRepository: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for i, condition in ipairs(obj.status.conditions) do
        if condition.type == "Ready" then
          if condition.status == "True" then
            hs.status = "Healthy"
          elseif condition.status == "False" then
            hs.status = "Degraded"
          else
            hs.status = "Progressing"
          end
          hs.message = condition.message
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for Ready condition"
    return hs
  # …repeat for ForgejoInstance, ForgejoUser, ForgejoUserKey,
  # ForgejoOrganization, ForgejoTeam, ForgejoWebhook, ForgejoDeployKey,
  # ForgejoOAuth2Application, ForgejoAccessToken, ForgejoNamespacePolicy.

Every kind uses the same script, the health.lua key of that ConfigMap.

NextCopy Anchor Link

Migrate an existing Forgejo.