public/security/gitea-incident-response.md

# When a Git Clone Found a Cryptominer: Investigating and Rebuilding a Compromised Gitea Server

Sometimes an incident announces itself through an alert, an IDS rule, or an unexpected spike in CPU usage.

This one started with a failed `git clone`.

While cloning a repository from a self-hosted Gitea instance, Git returned an unusual error:

```text
remote: /var/tmp/.gitea_cron_health: 6: Syntax error: "fi" unexpected (expecting "then")
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
```

A shell script in `/var/tmp` should obviously have had nothing to do with `git-upload-pack`.

That was the beginning of the investigation.

## The Environment

The affected service was a small self-hosted Gitea installation running inside an LXC container on Debian.

The architecture was deliberately simple:

```text
Internet
   |
 nginx reverse proxy
   |
 Gitea LXC
   |
 MariaDB LXC
```

Gitea ran as an unprivileged `git` system user. MariaDB lived in a separate container and the Gitea database account had privileges only on the Gitea database.

This separation turned out to be useful later.

The Gitea installation was also behind the current release line. Once compromise was suspected, upgrading the existing instance was no longer considered sufficient. The goal changed from "repair the service" to "determine what happened, preserve useful evidence, and rebuild from trusted components."

## First Discovery

Inspection of `/var/tmp` revealed several suspicious files associated with the `git` account:

```text
.gitea_cron_health
.wp_s2_cron
.sys_health_s3
.sys_health_s3.json
```

There were also PID/state files associated with them.

The largest file was approximately 6.4 MB and turned out to be a cryptocurrency miner.

Its configuration referenced Monero mining infrastructure and identified the worker with a name related to the compromised service.

More importantly, two watchdog processes were still alive:

```text
git ... sh -c while sleep 180; do /var/tmp/.gitea_cron_health; done
git ... sh -c while sleep 180; do /var/tmp/.wp_s2_cron; done
```

Both were running as the Gitea service account.

Their working-directory information also pointed into a Git repository managed by Gitea.

At this point the question was no longer whether the service had been compromised.

It had.

The questions became:

1. What level of access did the attacker obtain?
2. Was persistence limited to the Gitea account?
3. Was the host or container root compromised?
4. What secrets were exposed?
5. Could any existing server-side data still be trusted?

## Preserving Evidence Before Cleanup

Before deleting anything, the suspicious artifacts were copied to a separate incident directory.

SHA-256 hashes were calculated:

```text
58f9fcdfdba2354886df91263e8f03b9c3ee844db06bd7180fdd1b47bc6464ed  .gitea_cron_health
d2fcf28897ddc2137141d838b734664ff7592e03fcd467a433a51cb4976b4fb1  .sys_health_s3
2fd258697740454ddcbead6544a3e7c5ef5f529876827e995e6030280a006f4a  .sys_health_s3.json
a02834bd2edd207663c5d70af04644924c0dabd2cb09b8650aa492f0c4d95aba  .wp_s2_cron
```

Filesystem timestamps placed creation of the suspicious files within a narrow time window.

That gave us a point around which to search application logs.

This was an important lesson from the incident: deleting malware immediately may feel satisfying, but a few minutes spent preserving hashes, timestamps, process information, and logs can make the subsequent investigation dramatically easier.

## The Interesting Part of the Gitea Logs

Around the same time as the suspicious files appeared, Gitea logs showed an unusual sequence of HTTP requests.

The sequence included requests resembling:

```text
GET  /api/v1/version
GET  /api/v1/repos/search
GET  /<owner>/<repository>
POST /<owner>/<repository>
POST /api/internal/manager/add-logger
POST /api/internal/manager/remove-logger/...
```

Shortly afterward, Git operations against the same repository appeared.

One particularly unusual error was:

```text
BUG: upload-pack.c:254: packfile_uris requires sideband-all
```

Weeks later, the failed clone that triggered the investigation produced the same Git-side failure pattern while the malicious watchdog infrastructure was still present.

That correlation is significant.

It is not, however, enough to claim that a particular Gitea vulnerability or API endpoint was definitively the initial access vector.

That distinction matters.

The evidence demonstrates:

- attacker-controlled code executed as the Gitea `git` user;
- suspicious Gitea HTTP/API activity occurred around the creation time of the malware;
- Git repository operations were involved in the same time window;
- the compromised service was running an outdated Gitea release;
- similar Git failures occurred again when the malicious infrastructure was triggered.

It does not provide enough evidence to reconstruct every step of the exploit with certainty.

A useful incident report should distinguish evidence from hypothesis.

## Internet Background Noise

The logs also contained unrelated hostile traffic.

For example, later requests attempted SQL-injection-like payloads against application parameters and received HTTP 500 responses.

Those requests demonstrated that the public service was being actively scanned, but they did not demonstrate successful SQL injection.

This is another easy trap during incident response: once compromise is known, every ugly request in a web log starts looking like "the attack."

Public services receive enormous amounts of automated hostile traffic.

Temporal correlation and exploit-looking input are not proof of successful exploitation.

## How Far Did the Compromise Reach?

The malicious processes ran as:

```text
git
```

No evidence of root-level persistence was found during inspection of the container's cron configuration, systemd services, and other common persistence locations.

That does **not** prove that root compromise never occurred.

It means only that the investigation found no evidence supporting it.

There was, however, one consequence we could establish very clearly.

The Gitea service account could read:

```text
/etc/gitea/app.ini
```

Testing it directly confirmed that the `git` user had read access.

That configuration contained credentials and application secrets, including the database password and Gitea internal cryptographic tokens.

Therefore the correct security assumption was:

> Any secret readable by the compromised Gitea process must be considered compromised.

Whether the attacker actually extracted every one of those values was impossible to establish.

Operationally, that distinction did not matter.

They all had to be rotated.

## Database Isolation Helped

The MariaDB server ran separately from Gitea.

The Gitea database account had privileges only on:

```text
gitea.*
```

It did not have administrative privileges over the MariaDB server or unrelated databases.

This did not prevent the Gitea compromise, but it reduced what possession of the Gitea database credential could directly authorize.

This is a good example of why boring least-privilege configuration matters.

Most of the time it feels like administrative housekeeping.

During an incident it becomes a containment boundary.

## Repair or Rebuild?

At this point there were two possible approaches.

The first was to remove the miner, upgrade Gitea, rotate credentials, inspect the filesystem, and continue using the container.

The second was to destroy the service and rebuild it.

I chose the second.

Once arbitrary code has executed inside an Internet-facing service, proving that every modified file, hook, executable, scheduled task, configuration file, and application-controlled object is trustworthy can cost more time than rebuilding the service.

So the recovery rule became:

> Preserve evidence. Preserve trusted source history. Preserve as little of the compromised server as possible.

## Destroying the Old Application Environment

A completely new Debian LXC container was created for Gitea.

The old container configuration was preserved only where infrastructure identity mattered, including its network configuration. Keeping the same MAC address allowed DHCP to return the expected internal address, which meant the existing reverse proxy configuration continued to work without unnecessary network changes.

A current Gitea release was installed from the official distribution and its published checksum was verified before installation.

A new `git` system account was created.

New application directories were created from scratch.

New Gitea secrets were generated.

The old `app.ini` was not copied.

The new configuration was written manually.

The service was then installed as a new systemd unit.

No application binaries, hooks, temporary files, indexes, queues, or repository directories from the compromised container were reused.

## The Database Was Rebuilt Too

The old Gitea database was dropped.

The old MariaDB accounts used by Gitea were removed.

The MariaDB administrative password was changed.

A new Gitea database was created with a new application password and restricted privileges.

On its first startup, Gitea created a fresh schema.

One interesting moment occurred during initialization. The log stopped after reporting:

```text
Current database has been altered to use collation "utf8mb4_0900_as_cs"
```

Initially this looked like a database initialization hang.

MariaDB showed otherwise.

The Gitea connection was sleeping rather than blocked, and the database already contained more than one hundred newly created tables.

A controlled restart then produced:

```text
ORM engine initialization successful!
Listen: http://0.0.0.0:3000
Starting new Web server
```

The rebuilt application was alive.

## Separating Administrative and Normal Accounts

The rebuilt instance uses a dedicated administrative account rather than giving the everyday development account administrator privileges.

Public registration was disabled:

```ini
[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW  = false
```

Public repositories remain readable without authentication, while arbitrary visitors cannot create accounts.

The default landing page was also changed to the repository explorer:

```ini
[server]
LANDING_PAGE = explore
```

None of these controls would have prevented every possible application vulnerability.

They simply reduce unnecessary exposure and keep administrative privilege separate from routine use.

## Recovering Git Repositories Without Trusting the Old Server

The most important recovery decision concerned repositories.

The server-side repository storage was not treated as authoritative.

Fortunately, normal local Git clones already contained the commit objects and branch history required to reconstruct the projects.

This is one of Git's nicest properties during this kind of failure.

A commit contains its original author and committer metadata, including timestamps. Pushing an old commit to a new server does not recreate it with today's date.

For example, a repository might contain:

```text
main
feature-a
feature-b
```

The recovery was simply:

```bash
git push origin --all
git push origin --tags
```

The repository was first created empty on the new Gitea instance.

After pushing and fetching again:

```bash
git fetch --prune
git branch -a
```

all expected remote branches appeared.

No old server-side Git hooks were copied.

No old bare repository directories were restored.

Only objects reachable from trusted local clones were pushed into newly created repositories.

## Automating Repository Recovery

Once one repository had been successfully restored, the process was automated.

The recovery script recursively searched a directory tree for `.git` directories.

For every repository it:

1. verified that an `origin` existed;
2. used the existing remote URL rather than deriving one from the local directory name;
3. pushed all local branches;
4. pushed all tags;
5. reported success or failure;
6. continued processing the remaining repositories.

Using the existing `origin` was important because repository ownership and paths were not necessarily identical to the local directory structure.

The script deliberately avoided:

```text
git push --mirror
```

A mirror push is powerful, but it also mirrors deletions and every reference namespace. For recovery from ordinary working clones, explicitly pushing branches and tags was the safer operation.

## The 600 MB Surprise

One repository contained large binary trace data.

Its recovery failed with:

```text
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
...
Writing objects: 100% (150/150), 598.02 MiB
```

The problem was not Git or Gitea.

The nginx reverse proxy contained:

```nginx
client_max_body_size 200m;
```

The Git pack was approximately 598 MB.

Increasing the limit for the Gitea proxy location solved the problem:

```nginx
location / {
    proxy_pass http://gitea-internal:3000;

    client_max_body_size 2g;
    proxy_read_timeout 300s;

    ...
}
```

After validating and reloading nginx, the same repository pushed successfully.

Sometimes the final obstacle in a security incident is not security at all. It is a configuration value written years ago with a comment claiming it supports "large pushes."

Apparently, "large" is a relative term.

## What Was Recovered — and What Wasn't

Git history survived because it existed in distributed local clones.

That included:

- commits;
- commit graph;
- author information;
- author dates;
- committer dates;
- local branches;
- remote-tracking information still present in the clones;
- tags present locally.

Some Gitea-specific metadata did not survive the clean rebuild unless it existed elsewhere:

- issues;
- pull requests;
- comments;
- stars and watchers;
- repository creation timestamps;
- server-side activity events;
- some repository settings;
- server-side hooks and other application metadata.

This tradeoff was intentional.

The goal was not to reproduce every byte of the compromised system.

The goal was to recover trusted development history without importing potentially compromised server state.

## Backups: The Lesson After the Lesson

Once recovery was complete, the obvious final question was:

What happens if this needs to be done again?

The infrastructure already had separate backup storage, but the incident highlighted the value of having complete recoverable images of the service containers in addition to application-level data.

The resulting backup strategy became:

```text
Host
 ├── Gitea LXC
 ├── Database LXC
 ├── Other service LXCs
 └── configuration
          |
          +----> offline / separately managed backup storage
```

For consistency, containers containing databases can be stopped during a full filesystem backup, then restarted afterward.

A simple backup procedure can therefore:

1. record which containers are currently running;
2. stop those containers cleanly;
3. archive each complete LXC filesystem and configuration;
4. store the backups on separate storage;
5. restart only the containers that had been running before the backup;
6. retain multiple generations rather than overwriting the only known-good copy.

Snapshots are useful for rollback.

They are not backups when they live on the same storage as the thing they are supposed to protect.

## Lessons Learned

The most useful lessons from this incident were not particularly exotic.

**An application compromise does not automatically mean a root compromise.**  
Determine what the evidence supports instead of immediately assuming either the best or worst case.

**A compromised service account means its readable secrets are compromised too.**  
Application configuration files deserve the same threat-model attention as executable files.

**Least privilege matters after prevention has already failed.**  
A database account restricted to one database cannot magically become a database administrator just because its password leaks.

**Do not destroy evidence before understanding what you have found.**  
Hashes, timestamps, process state, working directories, and logs are cheap to preserve and expensive to reconstruct later.

**Do not confuse hostile scanning with successful exploitation.**  
Internet-facing logs are full of attacks. Evidence of an attempted exploit is not evidence that it worked.

**Do not overstate attribution or root cause.**  
In this incident, the timing and application activity strongly suggest a Gitea-mediated compromise, but the available evidence does not justify claiming a specific exploit chain as proven fact.

**Distributed version control is also a resilience mechanism.**  
Local Git clones were sufficient to reconstruct the important part of the repositories without trusting the compromised server.

**Rebuilding can be cheaper than proving cleanliness.**  
When an Internet-facing application has executed attacker-controlled code, a fresh operating environment with rotated credentials can be easier to reason about than an extensively "cleaned" machine.

**Backups need a trust boundary.**  
A snapshot beside the original data protects against mistakes. A separate backup protects against losing the system itself.

## Conclusion

The incident started with a syntax error from a shell script that should never have been involved in a Git clone.

It ended with a new operating system container, a new Gitea installation, a new database, new credentials, new application secrets, reconstructed repositories, tightened configuration, and a better backup strategy.

There is still one unanswered question: the exact initial exploit path.

That is acceptable.

Incident response does not always produce a perfect forensic narrative. Logs expire, evidence disappears, and attackers do not leave documentation.

The objective is to establish what can be established, clearly mark what cannot, contain the damage, remove untrusted state, recover trusted data, and leave the system in a better condition than before.

In this case, the most valuable artifact was not the miner binary.

It was the reminder that a failed `git clone` can occasionally be a security alert.