FSCI Service Status

[Resolved] Jitsi Meet Downtime notice - From 1630 to atleast 1650 IST on 18th June

Wed, 17 Jun 2026 20:31:00 +0000

Update (18/06/26) - Seems like there wasn’t much of a downtime. My monitoring picked 2 mins of network blips. Things OK.

Our VPS provider Contabo for meet.fsci.in has sent a maintenance downtime notice today.

Maintenance window starts at 18 June 2026, 16:30 IST and is expected to last at least 20 minutes. Expect Jitsi Meet to be unavailable during this time.

[Resolved] Jitsi Meet migration notice - Downtime 2100 to 2200 on 11th April

Fri, 10 Apr 2026 10:24:00 +0000

Update (11/04 - 2200) - All seem alright. Closing.

Update (11/04 - 2122) - Installation has been done. https://meet.fsci.in UP. Testing required.

We’re going to migrate Jitsi Meet from Docker package to native Debian package. Planned downtime - 2100-2200 on 11th April/Saturday.

OpenJDK21 wasn’t supported by Jitsi upstream, which Debian pushed. Latest update supports them now.

Details in https://codema.in/d/l3YJujeI/proposal-to-move-meet-fsci-in-to-india-to-my-mumbai-server- and https://github.com/jitsi/jitsi-meet/issues/16798

[Resolved] Tricky deployment maintenance on codema

Sun, 08 Mar 2026 10:50:00 +0000

I (asd) was doing a routine docker compose {pull, down, up -d} upgrade on loomio.

The warning to get rid of

Ever since the server upgrade in October 2025, we’ve been having docker warning us the following:

WARN[0000] volume "loomio-deploy_certs" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
WARN[0000] volume "loomio-deploy_vhost" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
WARN[0000] volume "loomio-deploy_html" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume  
[+] Running 11/11 "loomio-deploy_acme" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume

My hypothesis is that this was because we upgraded docker-compose plugin and the newer version isn’t detecting the older version’s volumes as its own.

Since these volumes seemed like recreatable data, I decided to remove those volumes and let docker compose recreate them (and start managing them as non-external).

docker volume rm loomio-deploy_certs loomio-deploy_vhost loomio-deploy_html loomio-deploy_acme

Unrecognized name

After doing this and restarting the service, codema.in was giving a 502 error in the browser.

It was tricky to debug this as docker compose ps showed the app was running healthily. docker compose logs also didn’t show any errors.

Here’s a brief aside on how the server is set up. It is like caddy -> exposed ports on docker -> nginx-proxy (inside docker) -> loomio app (inside docker). The Caddyfile config looks like:

http://codema.in, http://hocuspocus.codema.in {
        reverse_proxy http://127.0.0.1:8088
}

codema.in, hocuspocus.codema.in {
        reverse_proxy https://127.0.0.1:4443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
}

Ports 8088 and 4443 are exposed from docker with nginx-proxy handling those.

So, I looked at caddy logs with sudo journalctl -u caddy -f and it showed something like remote error: tls: unrecognized name when trying to connect to nginx-proxy.

I assumed that nginx-proxy is somehow not matching it to the right domain and tried fiddling with environment variables being passed to nginx-proxy. It didn’t help.

In nginx-proxy’s README there’s a section talking about missing certificates causing unrecognized name alert. So I checked inside the loomio-deploy_certs whether the certificates were actually present. And they were. The nginx-proxy logs also showed lines like the following which suggested that certs were available to nginx-proxy.

"ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/nginx/certs/codema.in.crt"
"ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/nginx/certs/hocuspocus.codema.in.crt"

Then I tried doing curl -H 'Host: codema.in' https://127.0.0.1:4443 and it was giving a similar error: error:0A000458:SSL routines::tlsv1 unrecognized name.

An issue on a different nginx-proxy-manager gave me the idea that this could be because nginx-proxy is using SNI for identifying server.

Letting caddy reverse proxy use SNI

So, with SNI, it is not enough to pass a header for the original host (which caddy does by default), but also pass tls_server_name option as follows:

codema.in, hocuspocus.codema.in {
        reverse_proxy https://127.0.0.1:4443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                        tls_server_name {host}
                }
        }
}

With that set, codema.in started working again.

Websocket error

But as always, I check browser console for any errors and I saw Firefox can’t establish a connection to the server at wss://codema.in/cable.

The request to /cable was getting a 502.

First I started wondering whether I should be configuring caddy to handle “wss” scheme. But on reading about websockets I realized it’s probably going over HTTP itself and caddy should be already working without any configuration.

Aside: /cable was introduced recently in 3.0.20 version of loomio which I upgraded loomio to 2 days before on Friday. There’s a very good chance that /cable was broken even before I started today’s maintenance. But unfortunately I did not know whether it was the case or not.

Again I started looking for logs from inside out. Found no log in docker compose that was useful. Eventually in caddy logs, I found something like:

http2: invalid Upgrade request header: ["websocket"]

Fortunately there was an issue on caddy repository about the same and turns out it was something to do with the HTTP version used for backend requests. There was a fix available which was released in 2.11 of caddy (and we were on 2.10).

So I did apt update and apt upgrade hoping to see the problem go away

Error 503 - Service temporarily unavailable

Well. What?

Now there was an nginx error page showing up saying the service was unavailable.

Again I started looking at all logs from inside out. Nothing had changed with the loomio app, it was running healthy. The nginx-proxy logged showed nothing except a 503. The caddy logs showed nothing either. What?

So I went inside the nginx-proxy container with docker compose exec nginx-proxy bash and ran curl https://codema.in/dashboard --resolve codema.in:443:127.0.0.1 (this is the way to do SNI with curl) and it was responding fine. Next, I looked at the nginx configuration and saw that there was a default server (in addition to codema.in server) which returned 503 when matched. I changed the return code to 504, did nginx -s reload and refreshed the browser and started seeing 504.

SNI is again failing

After breaking the glass I was drinking tea in I was like the only thing that changes is caddy version, what if there’s something new with caddy and SNI?.

So I looked at caddy releases and turns out the latest release 2.11.2 was from 2 days ago and had this highlight:

Reverse proxy got a lot of love with certain edge cases related to PROXY protocol, health check port, and closing body on retries. Dynamic upstreams are now tracked which enables passive health checking.

Convinced that this was a caddy issue, I immediately downgraded caddy apt install caddy=2.10.2 and the site was loading again.

But what about /cable?

In the issue about websocket error there was a workaround suggested, to set http version to 1.1 like:

                transport http {
                        tls
                        tls_insecure_skip_verify
                        tls_server_name {host}
                        versions 1.1
                }

Indeed that works.

But I didn’t like fixing the version like this. So I digged in the caddy release notes and discovered that in caddy 2.11.1 they changed the default behavior of reverse_proxy to pass host as upstream_hostport.

So eventually, the setting that works for now is:

codema.in, hocuspocus.codema.in {
        reverse_proxy https://127.0.0.1:4443 {
                header_up Host {host}
                transport http {
                        tls
                        tls_server_name {host}
                }
        }

}

[Resolved] Postgres version update for codema.in

Thu, 29 Jan 2026 12:00:00 +0000

Postgres in the docker-compose of codema.in was at version 9.6. We went to version 18.1. Steps are documented below.

Loomio was updated from stable (3.0.9) to latest (3.0.17)

Postgres upgrade steps

Stopping codema

docker compose down

Starting only db for dump

docker compose up -d db

Taking a dump

docker exec loomio-db pg_dump --clean --create loomio_production -U postgres > db.sql

Stop the db after dump

docker compose down

Moving the old data directory to new location (also makes for a backup)

sudo mv pgdata pgdata-backup

Upgrading postgres

vim .env
# Note the current version (9.6) and edit the postgres container version in .env to 18 
docker compose pull
docker compose up -d db

Copying dump to new instance

docker cp db.sql loomio-db:/db.sql

Restoring dump inside new instance

docker exec -it loomio-db bash
# Once inside container
psql -U postgres -f /db.sql

Restarting services

docker compose down
docker compose up -d

[Resolved] wiki.fsci.in Upgrade and server maintenance

Tue, 28 Oct 2025 12:00:00 +0000

Several maintentance upgrades were done:

Server upgraded from bookworm to trixie
Loomio upgraded from 3.0.7 to 3.0.9
Mediawiki upgraded from 1.37.2 to 1.44.2

Alongside, some other changes were also done:

Removed jitsi related packages (including prosody) because meet.fsci.in was moved to a new server recently.
Removed tinc (no longer used?)
Moved from nginx to caddy (easier to configure)
Upgraded docker and docker-compose-plugin using docker.com package
loomio-deploy upstream now uses haraka (instead of mailin), we too. It seems working too

Upstream bugs filed:

Suggested .env APP_URL value causes CORS

[Resolved] Upgrade codema.in to 3.0.7 and fight spam

Sat, 11 Oct 2025 13:00:00 +0000

We upgraded to v3.0.7 of loomio.

Also we enabled SPAM_REGEX to block spammy email domains. There was a brief period where we blocked too many domains greedily. Now it should block only spam domains. If you’re unintentionally affected by this, please reach out to FSCI admins or Akshay.

[Resolved] Moving meet.fsci.in to IN

Thu, 02 Oct 2025 17:00:00 +0000

Migration has been done. We’re in process of moving FSCI Jitsi meet service from Germany to India. We expect bit of downtime and service disruption during migraiton.

[Resolved] Peertube upgrade and host upgrade

Wed, 01 Oct 2025 22:00:00 +0000

The peertube instance was upgraded from v2.2.0 to v7.3.0

The underlying host was upgraded from buster to trixie

[Resolved] Mailman3 periodic update maintenance

Fri, 01 Nov 2024 18:07:00 +0000

[Resolved] Gitlab update - v17.0.8

Wed, 30 Oct 2024 06:15:00 +0000

Updating git.fosscommunity.in to v17.0.8 [Security Update]

[Resolved] Synapse Upgrade Notice

Sun, 27 Oct 2024 20:00:00 +0000

Sunday 27/10/2024 08:00 PM IST is scheduled for upgrading synapse. The process may take up to 6 hours and you may experience some downtime for matrix service during this period.

Update: Synapse is up now. Upgrade will continue on 28/10/2024 08:00 PM IST.

Update2: Synapse is upgraded to 1.85 now. Upgrade will continue on 29/10/2024 08:00 PM IST. Login is facing issues, giving “Incorrect username and/or password.” error.

Update3: Debugged login issue. Upgrade will be continued on 30/10/2024 08:00 PM IST.

Update4: Updated Worker configuration, homeserver configuration and nginx configuration. Systems seems to be stable now.

Update5: Matrix will be down for 2-3 hours at 11PM IST on 20/11/2024 for synapse upgrade.

Update6: Closing notice on 10/02/2025.

[Resolved] Poddery Upgrade Notice

Sat, 12 Oct 2024 11:00:00 +0000

Saturday 12/10/2024 11:00 IST is scheduled for upgrading Poddery server. The process may take up to 6 hours and you may experience some downtime for poddery services during this period.

Update: The upgrade is still going on, it is expected to see intermittent downtime for poddery services on 12/10/24 and 13/10/24.

Update2: We are now on Debian 12, Bookworm. All services are successfully running.

[Resolved] Mailman3 periodic update maintenance

Sat, 06 Apr 2024 17:07:00 +0000

[Resolved] Gitlab update - v16.8.3

Sat, 09 Mar 2024 22:34:00 +0000

Updated git.fosscommunity.in to v16.8.3 [Security Update]

[Resolved] Mailman3 periodic update maintenance

Sat, 17 Feb 2024 17:47:00 +0000

[Resolved] Gitlab update - v16.4.4

Mon, 01 Jan 2024 19:17:00 +0000

Updated git.fosscommunity.in to v16.4.4

[Resolved] Gitlab update - v16.2.8

Sun, 17 Dec 2023 12:00:00 +0000

Updating git.fosscommunity.in to v16.2.8

[Resolved] meet.fsci.in Debian version update

Sat, 04 Nov 2023 23:43:00 +0000

We have upgraded the server to bookworm.

[Resolved] Mailman3 periodic update maintenance

Thu, 12 Oct 2023 11:20:00 +0000

[Resolved] Gitlab 16.0.8 Security update

Sat, 07 Oct 2023 02:11:00 +0000

Updated git.fosscommunity.in to v16.0.8

[Resolved] Gitlab 16.0.7 Security update

Sun, 16 Jul 2023 12:17:00 +0000

Updated git.fosscommunity.in to v16.0.7

[Resolved] Gitlab 15.10.8 Security update

Tue, 13 Jun 2023 17:40:00 +0000

Updated git.fosscommunity.in to 15.10.7 -> 15.10.8

[Resolved] Gitlab 15.10.7 feature update

Fri, 02 Jun 2023 07:30:00 +0000

Updated git.fosscommunity.in to 15.9.8 -> 15.10.7

[Resolved] Gitlab 15.9.8 feature update

Mon, 29 May 2023 11:40:00 +0000

Updated git.fosscommunity.in to gitlab 15.8.5 (bullseye) -> gitlab 15.9.8 (bookworm)

[Resolved] Mailman3 periodic update maintenance

Sun, 23 Apr 2023 11:35:00 +0000

[Resolved] Gitlab 15.8.5 security update

Sun, 09 Apr 2023 12:32:00 +0000

Updated git.fosscommunity.in to v15.8.5

[Resolved] Gitlab 15.8.4 security update

Thu, 23 Mar 2023 23:02:00 +0000

Updated Gitlab to v15.8.4

[Resolved] Gitlab 15.7.8 security update

Wed, 08 Mar 2023 23:02:00 +0000

Updated Gitlab to version 15.7.8

[Resolved] Poddery Maintenance Notice

Sun, 19 Feb 2023 21:00:00 +0000

Sunday 19/02/2023 21:00 IST is scheduled for updating synapse to the latest release. The process may take up to 1 hour and you may experience some downtime for poddery.com during this period.

UPDATE: poddery.com is now running synapse v1.77.0.

[Resolved] Gitlab 15.6.6 security update

Sat, 11 Feb 2023 22:02:00 +0000

Updated Gitlab to version 15.6.6

[Resolved] Gitlab 15.5.7 security update

Tue, 10 Jan 2023 23:40:00 +0000

Updated Gitlab to version 15.5.7

[Resolved] Gitlab 15.5.4 security update

Sun, 01 Jan 2023 17:59:00 +0000

Updated Gitlab to version 15.5.4

[Resolved] Poddery Database Resizing

Sat, 31 Dec 2022 00:00:00 +0000

Poddery server maintainance from Saturday 31/12/2022 00:00 IST for database partition extension. The process may take upto 1 hour depending on the maintenance required and you may experience some downtime during this period.

[Resolved] Mailman3 EMERGENCY maintenance

Fri, 25 Nov 2022 21:00:00 +0000

[Resolved] Gitlab Migration

Sun, 20 Nov 2022 21:30:00 +0000

We’ll be migrating our gitlab instance to a new server.

[Resolved] Gitlab 15.4.2 feature update

Fri, 28 Oct 2022 15:36:40 +0000

Updated GitLab to v15.4.2

[Resolved] Poddery Maintenance Notice

Mon, 24 Oct 2022 23:00:00 +0000

Poddery server has to be checked for potential dirve-failure related issues. Expecting a downtime of 1 hour starting from 11:00 PM IST on 24/10/2022, Monday.

UPDATE [11:00 PM, 24th]: It is confirmed that we’re facing serious disk errors. All poddery services (diaspora, xmpp and matrix) will be down until a further announcement to take care of this issue.

UPDATE [01:30 PM, 31st]: Replaced the faulty disks and all services are up and running now. Thanks for your patience and cooperation.

[Resolved] Codema Migration Notice

Sun, 16 Oct 2022 20:00:00 +0000

Codema service is being migrated to a new server. Expecting a downtime of around 2 hours.

UPDATE [10 PM, 16th]: There’ll be more downtime extending upto 3 hours.

UPDATE [12 AM, 17th]: Migration couldn’t be completed as it needs more time than anticipated. Service is still up and running on the old server now.

UPDATE [2 PM, 17th]: Successfully migrated from Scaleway to Hetzner. Loomio (codema.in) and Jitsi (meet.fsci.in) are now running on the same server.

[Resolved] Poddery Matrix and XMPP Restart Notice

Thu, 13 Oct 2022 22:00:00 +0000

Poddery Matrix and XMPP services will be restarted on 14/10/2022 at 10:00 PM IST as part of regular maintenance.

[Resolved] Gitlab 15.3.4 security update

Fri, 07 Oct 2022 22:13:20 +0000

Updated Gitlab to version 15.3.4

[Resolved] Gitlab 15.3.2 feature update

Sat, 10 Sep 2022 17:47:40 +0000

Updated Gitlab to version 15.3.2

[Resolved] Gitlab 15.2.3 security update

Sat, 27 Aug 2022 22:22:20 +0000

Updated Gitlab to version 15.2.3

[Resolved] GitLab to v15.2.2

Sat, 20 Aug 2022 00:25:10 +0000

Updated GitLab to v15.2.2

[Resolved] Gitlab 15.1.4 update

Sun, 14 Aug 2022 22:20:00 +0000

Updated Gitlab to 15.1.4

[Resolved] Gitlab 15.0.4 major feature update

Sat, 23 Jul 2022 19:44:00 +0000

Updated Gitlab to 15.0.4 (major update from 14.10.5)

[Resolved] Mailman3 maintenance

Tue, 19 Jul 2022 08:22:00 +0000

Adding hcaptcha to avoid spam subscriptions.

[Resolved] Codema Maintenance Notice

Sun, 17 Jul 2022 14:00:00 +0000

Sunday 17/07/2022 14:00 IST is scheduled for testing loomio migration to new server. The process may take up to 4 hours and you may experience some downtme for codema.in during this period.

Update: The process started later than planned and ended without any downtme.

[Resolved] Gitlab 14.10.5 security update

Fri, 08 Jul 2022 15:49:15 +0000

Updated Gitlab to v14.10.5

[Resolved] Gitlab 14.9.5 security update

Sat, 11 Jun 2022 18:26:00 +0000

We have updated gitlab to v14.9.5

[Resolved] Codema.in Upgrade

Mon, 16 May 2022 21:33:00 +0000

Codema has not been updated in a long time due to various issues in migration, some of them related to upstream.

We’ve been able to find a migration path of least resistance and doing this upgrade, however risky it maybe, leads to good because it becomes increasingly difficult to do an upgrade the more we wait.

Update

We went from 2.0.108 to 2.12.1 which means a lot many things have changed.

If you find anything off, do let asd know.

Issues filed upstream as a result of this upgrade:

Backstory

The migration was blocked earlier by an issue with automatic ID generation for the ‘stances’ table. The sequence ‘stances_id_seq’ believed that the next number was somewhere around 30k. But for some unknown reason there already were a few entries in this table with IDs upto 80k.

This was identified by Harish and fixed by Akshay with the following psql command:

select setval('stances_id_seq', 879505, false);

This manipulated the sequence to allow for uninterrupted ID generation. (To arrive at the number 879505 one just had to do a select id from stances order by id desc limit 1; to get the highest id in the table).

There were further issues in later migrations, but those were crossed with a bit of nudge from upsream (see github issues linked above).

What would have been a roadblock was lack of disk space. When we began the update, df -h showed

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        19G   16G  1.7G  91% /

Pirate Bady was quick to point out that there were some image backups that could be deleted to save space. docker system prune -a also helped in reclaiming space between each docker-compose pull.

The final path that was taken was

v2.0.108 (starting)
v2.4.2
v2.8.8
v2.11.13
~~v2.12.1~~ v2.13.1 (present stable)

The underlying loomio-deploy docker-compose has to be updated, but that can wait for another day.

[Resolved] Mailing List usual update

Wed, 11 May 2022 12:50:00 +0000

Usual maintenance.

[Resolved] Gitlab 14.8.6 security update

Wed, 04 May 2022 22:23:00 +0000

We have Updated Gitlab to v14.8.6

[Resolved] GitLab to v14.8.5

Fri, 08 Apr 2022 22:13:25 +0000

We have updated Gitlab to v14.8.5

[Resolved] GitLab v14.7.7 security+feature update

Sat, 02 Apr 2022 23:45:00 +0000

We have updated Gitlab to v14.7.7

[Resolved] Poddery server restart

Sat, 02 Apr 2022 21:40:00 +0000

We’re have successfully restarted the server.

[Resolved] Prosody v0.11.13 update

Mon, 21 Mar 2022 21:45:00 +0000

We updated Prosody to v11.13

[Resolved] GitLab v14.6.5 security update

Wed, 02 Mar 2022 19:00:00 +0000

We are updating Gitlab to v14.6.5

[Resolved] Mailing List usual update

Mon, 28 Feb 2022 10:11:00 +0000

Updating mailman dependencies and usual maintenance.

[Resolved] GitLab v14.6.4 feature+security update

Sun, 06 Feb 2022 21:45:05 +0000

We have updated Gitlab to v14.6.4

[Resolved] Jitsi Meet Server Migration

Mon, 31 Jan 2022 21:52:00 +0000

We have migrated meet.fsci.in server from hetzner CPX31 to CPX21 instance.

[Resolved] Mailing List Maintenance

Sun, 16 Jan 2022 18:44:00 +0000

Server was down due to issues related payment and storage.

[Resolved] Diaspora v0.7.15.0, Prosody v0.11.12-1, Matrix-Synapse v1.49.0-1 update

Sat, 15 Jan 2022 22:35:00 +0000

We have updated

Diaspora to v0.7.15.0
Prosody to v0.11.12,
Matrix-Synapse to v1.49.0-1
System update.

[Resolved] GitLab v14.5.3 security update

Fri, 14 Jan 2022 22:34:05 +0000

We have updated Gitlab to 14.5.3

[Resolved] Peertube Maintenance

Fri, 14 Jan 2022 14:00:00 +0000

Service was down due to lack of sufficient storage space.

[Resolved] GitLab v14.4.4 security update

Fri, 10 Dec 2021 21:16:00 +0000

We have updated Gitlab to v14.4.4

[Resolved] Jitsi Meet 2.0.6689 update

Thu, 09 Dec 2021 23:40:00 +0000

We have updated Jitsi Meet to v2.0.6689. Breakout room support has landed.

[Resolved] GitLab to v14.4.2

Sun, 28 Nov 2021 21:10:00 +0000

We have updated Gitlab to v14.4.2

[Resolved] Poddery Bullseye update

Sat, 27 Nov 2021 20:50:00 +0000

We have updated poddery.com server to bullseye.

[Resolved] Poddery Matrix Synapse 1.47.1

Tue, 23 Nov 2021 21:45:00 +0000

We’re updating poddery’s matrix-synapse to 1.47.1

Update: Service is operational now after 4 hours of downtime

[Resolved] Jitsi Meet 2.0.6433-1 update

Fri, 19 Nov 2021 21:52:00 +0000

We have updated jitsi meet to v2.0.6433-1 release.

[Resolved] GitLab to v14.3.4

Fri, 19 Nov 2021 21:23:00 +0000

We have updated gitlab to 14.3.4 feature release.

[Resolved] Updating GitLab to v14.2.6 and PostgresSQL to v13

Sat, 06 Nov 2021 23:00:00 +0000

We have updated gitlab to 14.2.6 security and feature release. We have also updated postgres to 13 from 11.

[Resolved] Renew Hosting

Fri, 22 Oct 2021 03:54:00 +0000

[Resolved] Poddery Matrix Synapse 1.44.0

Wed, 06 Oct 2021 22:30:00 +0000

We updated poddery’s matrix-synapse to 1.44.0

[Resolved] Update gitlab to the latest

Tue, 05 Oct 2021 22:29:00 +0000

Updated Gitlab to the latest version

[Resolved] Jitsi Meet 2.0.6293 update

Sat, 18 Sep 2021 23:40:00 +0000

We have updated Jitsi Meet to v2.0.6293.

[Resolved] Gitlab 14.0.10 feature update

Sat, 11 Sep 2021 22:50:00 +0000

We have updated GitLab to 14.0.10 feature update.

[Resolved] lists.fsci.in maintenance

Sat, 11 Sep 2021 20:14:00 +0000

We updated to new Debian release !

[Resolved] Codema Maintenance Notice

Sun, 05 Sep 2021 23:00:00 +0000

Sunday 05/09/2021 22:00 IST is scheduled for updating loomio. The process may take up to 1 hour and you may experience some downtime for codema.in during this period.

Update: The update was cancelled due to an error in database migration. The service was restored to older state after about 1 hour of downtime.

[Resolved] Jitsi meet server bullseye and 2.0.6173 update

Sun, 29 Aug 2021 18:50:00 +0000

We have updated Jitsi meet server to bullseye and updating Jitsi meet to v2.0.6173.

This Jitsi update features screensharing on 5/15/30 frames, which can help in video over screensharing play.

[Resolved] Poddery nginx configuration fixes

Fri, 20 Aug 2021 10:00:00 +0000

Details at https://diasp.in/posts/1877344

[Resolved] Intermittent service outage

Thu, 19 Aug 2021 14:30:00 +0000

There will be intermittent outages on gitlab while we are configuring Rack::Attack to prevent brute force attacks.

[Resolved] Gitlab 2FA fix update

Mon, 09 Aug 2021 23:40:00 +0000

We are fixing broken 2FA on our GitLab instance.

[Resolved] Gitlab 13.12.9 security update

Thu, 05 Aug 2021 22:35:00 +0000

We updated GitLab to 13.12.9 security release.

[Resolved] Gitlab 13.12.8 security update

Sat, 10 Jul 2021 22:23:00 +0000

We have updated GitLab to 13.12.8 security update.

[Resolved] Gitlab 13.12.6 security update

Sat, 03 Jul 2021 22:35:00 +0000

We have updated GitLab to 13.12.6 security update.

[Resolved] Poddery Matrix Synapse Update

Sun, 27 Jun 2021 14:00:00 +0000

Sunday 27/06/2021 14:00 IST is scheduled for updating synapse matrix server. The process may take upto 1 hour depending on the maintenance required and you may experience some downtime during this period.

Update: Synapse is on v0.36 now. This took a little more time than anticipated. But downtime was almost negligible as there were only a couple of restarts in between.

[Resolved] Update Debian 10 to 11

Sat, 26 Jun 2021 22:40:00 +0000

We updated Debian 10 (buster) to Debian 11 (bullseye).

[Resolved] Poddery Matrix Synapse Update

Sun, 20 Jun 2021 23:00:00 +0000

Sunday 20/06/2021 23:00 IST is scheduled for updating synapse matrix server. The process may take upto 1 hour depending on the maintenance required and you may experience some downtime during this period.

NOTE: Synpase not upgraded due to some technical issues. The currently running version 1.20.1 is very old and needs more time and care than anticipated. This will be fixed soon.

[Resolved] Gitlab 13.12.3 feature update

Sat, 19 Jun 2021 22:20:00 +0000

We have updated GitLab to 13.12.3 feature update.

[Resolved] Gitlab 13.11.5 security update

Fri, 04 Jun 2021 22:20:00 +0000

We have updated GitLab to 13.11.5 security update.

[Resolved] Jitsi Meet 2.0.5870 update

Sun, 16 May 2021 23:50:00 +0000

We have updated Jitsi Meet to v2.0.5870.

[Resolved] Storage issues

Tue, 04 May 2021 21:23:00 +0000

Storage capacity on the server hosting the GitLab instance exhausted and all services on the server failed. The primary culprit was the log directory ( /var/log/gitlab ) which was at 51 GB.

Truncating the log file still did not bring the service back online. The service was restored after multiple services including database server ( postgreSQL ) and caching server ( redis ) were restarted.

[Resolved] Gitlab 13.11.2 feature update

Mon, 03 May 2021 22:40:00 +0000

We have updated GitLab to 13.11.2 feature release.

[Resolved] Gitlab 13.10.4 security update

Wed, 28 Apr 2021 22:45:00 +0000

We have updated GitLab to 13.10.4 security update.

[Resolved] Jitsi Meet 2.0.5765 update

Thu, 22 Apr 2021 22:41:00 +0000

We have updated to Jitsi Meet to 2.0.5765.

[Resolved] Gitlab 13.9.6 security update

Fri, 16 Apr 2021 22:55:00 +0000

We have updated GitLab to 13.9.6 security update.

[Resolved] Gitlab 13.9.5 feature update

Sun, 04 Apr 2021 15:14:00 +0000

We have updated gitlab to 13.9.5 feature + security release.

[Resolved] Mailman3 instance maintenance

Mon, 29 Mar 2021 12:21:00 +0000

DKIM Keys were added for lists.linuxdelhi.org. Systemd failed on upgrade which broke the services. Systemd upgraded and restored services. Mailman issues with upgrade resolved. Mailman3 and Django packages were upgraded to latest. Rspamd was upgraded. Unbound DNS resolver was setup to handle dns queries mainly for rspamd.

[Resolved] videos.fsci.in maintenance

Sun, 28 Mar 2021 12:00:00 +0000

Peertube has been upgraded to 3.0 and storage has been increased

[Resolved] Peertube storage upgrade

Mon, 22 Mar 2021 00:30:00 +0000

Peertube server needs to be restarted for upgrading storage space. There’ll be a small downtime on 22/03/2021 (Monday) between 00:30 and 01:00.

[Resolved] Gitlab 13.8.6 security update

Fri, 19 Mar 2021 20:14:00 +0000

We are updating gitlab to 13.8.6 security release.

[Resolved] Gitlab 13.8.5 feature update

Wed, 10 Mar 2021 11:30:00 +0000

We updated gitlab to 13.8.5 feature release.

[Resolved] Gitlab 13.7.8 security update

Sat, 06 Mar 2021 17:30:00 +0000

We updated gitlab to 13.7.8 feature + security release.

[Resolved] Gitlab 13.6.7 security update

Sat, 13 Feb 2021 21:45:00 +0000

We updated gitlab to 13.6.7 security release.

[Resolved] codema.in down due to payment failure

Sat, 13 Feb 2021 18:00:00 +0000

[Resolved] Gitlab 13.6.6 security update

Fri, 05 Feb 2021 00:55:00 +0000

We updated gitlab to 13.6.6 feature + security release.

[Resolved] codema.in scheduled maintenance by scaleway

Tue, 26 Jan 2021 20:30:00 +0000

A notice of maximum 1h downtime is issued by Scaleway due to their migration maintenance. See https://status.scaleway.com for more details.

[Resolved] Gitlab 13.5.7 security update

Mon, 18 Jan 2021 18:10:00 +0000

We updated gitlab to 13.5.7 feature + security release.

[Resolved] Gitlab 13.4.7 security update

Thu, 10 Dec 2020 00:50:00 +0000

We updated gitlab to 13.4.7 security release.

[Resolved] Gitlab 13.4.6 feature update

Thu, 03 Dec 2020 16:40:00 +0000

We updated gitlab to 13.4.6 feature release. We ran out of memory during the update, so it took more time than usual.

[Resolved] codema.in scaleway migration maintenance

Tue, 10 Nov 2020 16:30:00 +0000

scaleway completed the migration of codema vm to new physical hypervisor

[Resolved] Gitlab 13.3.9 feature + security update

Sun, 08 Nov 2020 13:20:00 +0000

We updated gitlab to 13.3.9 feature + security release.

[Resolved] Gitlab 13.2.10 security update

Mon, 12 Oct 2020 15:20:00 +0000

We updated gitlab to 13.2.10 security release.

[Resolved] Gitlab 13.2.8 security update

Thu, 03 Sep 2020 21:55:00 +0000

We updated gitlab to 13.2.8 security release. We also switched ruby on rails application server from unicorn to puma (improves performance).

[Resolved] Gitlab 13.2.6 security update

Thu, 20 Aug 2020 01:30:00 +0000

We updated gitlab to 13.2.6 security release.

[Resolved] Gitlab 13.1.6 security update

Sun, 09 Aug 2020 17:15:00 +0000

We updated gitlab to 13.1.6 security release.

[Resolved] git.fosscommunity.in: troubleshoot issues created by ruby update to 2.7

Tue, 14 Jul 2020 13:00:00 +0000

All repos showed 503 error when viewing via web interface. It was caused by an old version of bundler installed for ruby 2.5 not working with ruby 2.7. It was fixed by removing the old version of bundler and restarting gitaly service.

[Resolved] videos.fsci.in peertube instance migration

Sun, 12 Jul 2020 15:30:00 +0000

The peertube instance of fsci.in has been migrated to new server and is working fine.

[Resolved] codema.in sheduled maintenance

Sun, 12 Jul 2020 02:30:00 +0000

[Resolved] git.fosscommunity.in: ruby 2.7 update

Sat, 11 Jul 2020 14:20:00 +0000

We are updating ruby to version 2.7 on git.fosscommunity.in.

[Resolved] Gitlab 13.1.3 security update

Thu, 09 Jul 2020 23:50:00 +0000

We updated gitlab to 13.1.3 security release.

[Resolved] Gitlab 13.1.2 security update

Thu, 02 Jul 2020 18:00:00 +0000

We updated gitlab to 13.1.2 security release.

[Resolved] Gitlab 13.1 feature update

Tue, 30 Jun 2020 23:50:00 +0000

Tuesday 30/06/2020 starting from 23:50 to Wednesday 01/07/2020 01:30, we will be updating gitlab to 13.1 feature release.

[Resolved] Poddery Matrix Database Maintenance - Second Phase

Mon, 22 Jun 2020 23:30:00 +0000

The matrix synapse server was down for performing various database optimization operations such as removing empty rooms, compressing data, etc.

[Resolved] Poddery Matrix Database Maintenance

Mon, 22 Jun 2020 15:30:00 +0000

Monday 22/06/2020 starting from 15:30 to 18:30 we will work on some database maintenance issues, so you may experience some downtime between 1 to 3 hours. Services affected:

Matrix

Setup cstate for FSCI

Tue, 05 May 2020 21:12:00 +0000

Initial work on setting up cstate for fsci.in