Unable to setup a reliable REI3 instance

Julien

Hi there,
first off, I must say that I'm impressed with REI3 with all its features!

The issue
I'm down to about 8-10h trying to setting up an instance on a local server. Following the documentation, all went pretty well except that after every reboot of the server, the login page gets stuck on the "Waiting for server connection..." message and the console logs "WebSocket is closed before the connection is established." every few seconds. I just can't nail the reason why it happens. On each reboot, I go change some settings and it magically starts working again (until next reboot).
I feel that I'm close to find the solution to that seemingly random problem but I'm running out of solutions...

My setup

REI3 3.10 for Linux on Ubuntu Server 24.04 on a VM
Postgresql 16
Accessing VM through SSH and REI3 from an M1 mac
REI3 is listening on port 8080
REI3 is installed in the server's user's home folder (/home/rei3/r3).

More info
Launching REI3 with the -http flag doesn't solve the problem.

When running the commands systemctl status postgresql@16-main and netstat -tunelp | grep 5432, I can confirm that Postgres is active and listening on the right port.

When running the command systemctl status rei3, I can tell that REI3 is active and logging incoming request from the client browser on the mac.

Have anyone experienced this kind of problem? Any advice could help me figure out how to make REI3 work for good! 😅
Thanks in advance for any help!

gabriel

The setup sounds fine from what you describe. In general, if all services are running (r3, postgres) we need to take a look at network and client.

Do you use a proxy?
Do you have the same issues with another browser?

Some other points:

What changes do you do so that it suddenly works? Do you have an example?
When restarting the server, REI3 takes a bit of time to build its cache - depending on the system and installed apps, this can take even a full minute. In this case you would notice system load on the r3 and postgres processes until the cache is ready. While the server cache builds, clients (eg. browsers in this case) would still show "Waiting for server connection".

Julien

gabriel
Hi Gabriel. I just spend another 2 hours trying to figure out the issue. While I don't completely understand what's happening, I managed to narrow it down a bit.

It turns out that if I reboot the server machine while I'm logged in REI3 from a client browser, when the server is back running up, I won't be able to connect from the client anymore. The solution (which seems to work half the time):

is to stop rei3.service or shutdown the server
operate an "Empty cache and hard reload" on the client's browser
start rei3.service or boot the server machine back up
wait few 20 seconds to give rei3 some time to fully start

It feels like when a web page loose connection with the server, there are some data that remains in the cache and prevents it from being able to connect back after then. I'm not 100% sure about this because some times the steps above didn't fix the "Waiting for server connection..." issue.

Also, I'm not sure about what the setting sslSkipVerify in config.json means, but it seems that it needs to be set to true for the server to be accessible at all, with SSL activated.

To conclude, while I can manage to setup and access an REI3 instance, I still haven't found an answer to this topic's title on how to setup a reliable REI3 instance; one which clients could access without worry...

If there are any information that I could provide to help solve this, I'm willing to share.

Again, thanks for making this product freely accessible

Julien

Hi Gabriel and thanks for the quick reply. I haven't had the time to try out solutions yet but as soon as will have time, I'll share the results and possibly a solution on the forum.

Your help is much appreciated!

gabriel

To continue troubleshoot, please respond to my previous questions above. We´ve had some connection issues in the past when proxies were involved. Please let me know, if you have a proxy inbetween. If it´s a cache issue, it is important to know what browser you use and if you have the same issue with different browsers. As an example of why this is important, if you are using a self signed certificate, we know that some browsers can be picky and reject reconnection - then only a cache-reset helps so the user can re-confirm to trust the self-signed certificate.

Julien Also, I'm not sure about what the setting sslSkipVerify in config.json means, but it seems that it needs to be set to true for the server to be accessible at all, with SSL activated.

This setting relates to the database connection between r3 and your database server. It either works (if your server speaks SSL) or not. It has no influence on any client trying to talk to r3. If the DB connection fails, the r3 service will shutdown on its own.

I can offer a short teams call. Setting up r3 is usually simple enough - we should be able to quickly identify the issue. If you are interested, send us your contact details and we´ll find a time.

Julien

I have no proxy.
To my surprise, I don't seem to have the issue with Safari! Since the beginning I've been using Brave (Chromium). With Safari, I can leave the page on, reboot the server, and the page will automatically show the login inputs!

Brave gets stuck with a looping warning in the console: "WebSocket connection to '<URL>' failed: WebSocket is closed before the connection is established."
Like I mentioned yesterday, doing an "Empty cache and hard reload" doesn't help...

While it might be a certificate issue, I've had the same issue with the server running with -http flag two weeks ago (I could try to reproduce it if it could be helpful)...

UPDATE:
Perhaps, I should've mentioned that in Brave There's an error appearing once on page load. Here's a screenshot:

gabriel

Thank you for posting the screenshot and providing details. The error message is a known issue where a browser that has a user-initiated security exception (continue with access while accepting an invalid SSL certificate) does not reinitiate the connection because the previous security exception is forgotten.

This is an issue that some browsers suffer from. Modern browsers really do not wish you to access sites or apps with invalid SSL certificates. They allow you to accept the connection, but they are not happy about it - and some browsers fail to load resources, because they ignore or forget the previous security exception. This happens with r3 specifically, because we use a websocket connection that needs to be re-established if the connection to the server goes down for whatever reason.

Not every browser is affected, but I´ve seen this multiple times. The solution is to use a proper SSL certificate. Then there is no exception that a browser can ignore when it feels like it.

As to why you have issues running with -http, I cannot say without looking at it or error messages if there are any. Do you get any errors in the case of HTTP access?

Julien

I just ran a few tests in -http mode and the error message is very similiar:

It happens when I simply to a refresh on the page. On the screenshot we see that the message was sent 9 times and then the website went back on track by itself. So it's not that big of an issue...

About the Brave browser being unhappy with invalid certificates, I suppose that for my local testing I could remain on -http and when/if I'm ready to go into production, getting real ssl certificates will solve the issue?

gabriel

Yes, using a valid SSL cert permanently gets rid of that issue.

Though the connection issues via HTTP worry me. Because we do not know the reason, we do not know if this issue is going to be a problem later. A valid SSL cert will not help with this.

Do you get this issue with Safari and Brave or just one of your browsers? Do you have any addons/extensions active? If yes, does the issue persist when you disable them?

Julien

gabriel
Well that's annoying... I just tried to screen-record the issue but it seems to be gone; page refreshes without any issue!

I don't know what to say... Next time it happens I'll try to record it and test it with another browser. I could also run r3 in debug mode while redirecting the stdout to a log file.

gabriel

Debug mode is used for debugging and will flood your system with thousands of large log entries. I would strongly recommend not enabling this unless you are debugging the source code. An error that would cause the r3 service to crash would be printed.

But since the service keeps on trucking and the sympton is that a particular browser does not connect reliably, the issue is more likely not found within the service itself.

As mentioned before, if you can reproduce the issue, I offer to take a look. If I can see the issue live, I can likely tell you what broke.