Made a looking-glass clone for fun

ericls

Hi LET,

As a weekend project, I decided to give Deno another try and tried to build a looking glass with it. I'm not unhappy with the existing PHP one, just thought it would be a good project to try out a new tool. The goal is to have most of the visible functionalities as the PHP one, and I was really curious to see if it's possible to pack everything inside a single executable.

It kinda worked, but the way Deno "compiles" things is that it just bundles your code and then bundle that bundle with the Deno executable, which is fine, but the file size is not optimial, around 80MB...

It generates the config file on the first run if it doesn't exists, and warns about missing commands. The config file should be self-explanatory.

Demo: http://199.195.253.86/
Code: https://github.com/ericls/looking-glass
Download: https://github.com/ericls/looking-glass/releases

Feel free to take a look, fork it, play with it or don't care about it

yoursunny

It's the first Deno project I read, and it's surprising easy to understand (revision 5347815918cd1ac341d3b65b73aec72e9456cbb6).

I'm concerned on the performance of downloadable files.
Downloadable files on a looking glass is intended to be representative of network bandwidth.
Is crypto.getRandomValues(p) fast enough so that it would not become a bottleneck?

I see some attempt on preventing ping and traceroute to private addresses.
However, I can easily get around this by publishing an A record that point to a private address.

RateLimiter#clientCounter is only added to but never deleted from.
It would consume too much memory after many different IPs have accessed the socket, especially when the server is deployed on IPv6 endpoint.
You should delete a counter when it reaches zero, and not storing zero in the object.

ericls

@yoursunny said:
It's the first Deno project I read, and it's surprising easy to understand (revision 5347815918cd1ac341d3b65b73aec72e9456cbb6).

I'm concerned on the performance of downloadable files.
Downloadable files on a looking glass is intended to be representative of network bandwidth.
Is crypto.getRandomValues(p) fast enough so that it would not become a bottleneck?

I see some attempt on preventing ping and traceroute to private addresses.
However, I can easily get around this by publishing an A record that point to a private address.

RateLimiter#clientCounter is only added to but never deleted from.
It would consume too much memory after many different IPs have accessed the socket, especially when the server is deployed on IPv6 endpoint.
You should delete a counter when it reaches zero, and not storing zero in the object.

Thanks for you reply!

Good observations.
I had some of the same thoughts, except the one with private IP bypass, which is a good find.

Not sure about the performance of getRandomValues, but I read it on MDN that it should be fine. I don't have enough data to support it tho.

I think tho, that it's always insecure to run user submitted things on a non-sandboxed environment, not sure how to go about that...

FAT32

@ericls said:
Not sure about the performance of getRandomValues, but I read it on MDN that it should be fine. I don't have enough data to support it tho.

I havent read the code but from my understanding secure random usually involve reading from environment, devices without enough entropy might slow it down.

I think tho, that it's always insecure to run user submitted things on a non-sandboxed environment, not sure how to go about that...

Again, sorry that I havent read the code. In Node.js (which I believe is very similar to Deno), you can use VM to run them in separate context. But anything calling shell command is tricky to restrict, make sure you use whitelist approach instead of blacklist.

FAT32

Ok finally decided to go in front my laptop and read the code. I would say the code quality is decent Can be better but quite good.

Some comments:

---

Try not to use window. object to store global data but instead use a Singleton, it will make TypeScript hinting more accurate and easier to read.

---

If the setting includes single quote (eg. inside siteTitle), the following code will break:

https://github.com/ericls/looking-glass/blob/3bb51bdeadf562e323b36911e6b441ad3ec5fe63/src/index.tsx#L81-L90

The solution that we usually use is double encoding, not optimal for size but more secure. Also try to replace the character < into \u003c after encoding. Basically:

(Same for classes)

---

Your rate limiter only record for success request (eg. JSON.parse being valid), recommend moving it to the top to also record failed request.

---

isValidHost is not very safe but Deno seems to auto encode all the arguments before passing to shell so should be relatively safe.

---

Might also want to consider adding a timeout in the Deno code instead of relying on Shell to return. (It can be as easy as setTimeout or use Promise.race())

---

As mentioned by yoursunny, generating static files might be faster than randomly generate on the fly. You can generate it one-off when looking glass is first initialised/executed.

---

Ok feel like I am doing a code review now sorry - Anyway code looks clean, keep it up!

yoursunny

I did a quick benchmark.
Hardware: Xeon Gold 6240, 100Gbps connection.
Server runs in Docker container.

ericls/looking-glass v0.2.0:

$ curl http://192.168.94.4/file/1000MB.test > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  953M    0  953M    0     0   517M      0 --:--:--  0:00:01 --:--:--  516M

Caddy file server, NVMe drive, non-first load:

$ curl http://192.168.94.5/1000MB.bin > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1000M  100 1000M    0     0  1908M      0 --:--:-- --:--:-- --:--:-- 1904M

---

@FAT32 said:

@ericls said:
Not sure about the performance of getRandomValues, but I read it on MDN that it should be fine. I don't have enough data to support it tho.

I havent read the code but from my understanding secure random usually involve reading from environment, devices without enough entropy might slow it down.

Entropy is not a concern.

Deno crypto.getRandomValues is implemented with Rust thread_rng.fill().

Rust RNG is implemented with getrandom syscall.

getrandom(2) draws entropy from the urandom source that does not block for gathering entropy.

1gservers

Wouldn't it be fastest to serve the test files from a ramdisk?

Daniel15

So this has one of the small issues I had with DNSTools a while back: The ping tries to block pinging private networks (eg. 192.168.x.x or 10.x.x.x), but it doesn't block it if you use a hostname that resolves to a private IP (eg. try la03.int.d.sb).
You may also want to consider showing timeouts (no reply) via -O. The exact command I use on DNSTools is ping -i 0.5 -c 5 -O <ip>: https://github.com/Daniel15/dnstools/blob/master/src/DnsTools.Worker/Tools/Ping.cs#L41

Why are you using SIGKILL (process.kill(9)) rather than SIGTERM when reading stderr? https://github.com/ericls/looking-glass/blob/3bb51bdeadf562e323b36911e6b441ad3ec5fe63/src/socket.ts#L64

FAT32

@1gservers said:
Wouldn't it be fastest to serve the test files from a ramdisk?

I like this idea, basically what we actually need is just some of the bytes to be random enough. When the app first started, we can initialise 50% of available memory in the heap. After that, store some random generated bytes in the heap memory and serve them whenever there's request to test files. (In case the requested file is larger than available memory, just do a modulo)

FAT32

@Daniel15 said:
Why are you using SIGKILL (process.kill(9)) rather than SIGTERM when reading stderr? https://github.com/ericls/looking-glass/blob/3bb51bdeadf562e323b36911e6b441ad3ec5fe63/src/socket.ts#L64

Everytime I see SIGKILL / -9 it reminds me of this comic

Daniel15

@FAT32 said: Everytime I see SIGKILL / -9 it reminds me of this comic

aww poor SIGKILL'd penguins (...are those even penguins?)

aqua

Demo seems to be offline, if you'd like a free DDoS protected VPS to host it on, shoot me a PM

yoursunny

@1gservers said:
Wouldn't it be fastest to serve the test files from a ramdisk?

With kernel buffer cache, I used to expect only minor difference, except on first load.
A quick benchmark indicates that serving from a ramdisk increases download speed by 13% on second load.

Caddy file server, NVMe drive, first load and second load.
Before starting the HTTP server and testing first load, I executed sync; echo 3 > /proc/sys/vm/drop_caches to flush the buffer cache.

$ curl http://192.168.94.2/100G.bin > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100G  100  100G    0     0   879M      0  0:01:56  0:01:56 --:--:--  922M
$ curl http://192.168.94.2/100G.bin > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100G  100  100G    0     0  1703M      0  0:01:00  0:01:00 --:--:-- 1724M

Caddy file server, /dev/shm mount, first load and second load.
I flushed the buffer cache using the same procedure, but noticed that free -m command indicates more than 100GB still in the buffer cache, suggesting that files in /dev/shm persist in the buffer cache.

$ curl http://192.168.94.2/100G.bin > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100G  100  100G    0     0  1685M      0  0:01:00  0:01:00 --:--:-- 1632M
$ curl http://192.168.94.2/100G.bin > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100G  100  100G    0     0  1931M      0  0:00:53  0:00:53 --:--:-- 1912M

ericls

@FAT32 Thanks so much for reviewing
@yoursunny Thanks for benchmarking it!

I probably don't have time during the week to fix the issues but should be able to fix them on the weekend.

There's one really pretty bad bug on oak that crashes the server if there's no host header...
https://github.com/oakserver/oak/issues/386
Hopefully that can be solved as well.

Levi

For entropy (if any issues) use haveged. Such simple application often overlooked.

ericls

@Daniel15 said:
So this has one of the small issues I had with DNSTools a while back: The ping tries to block pinging private networks (eg. 192.168.x.x or 10.x.x.x), but it doesn't block it if you use a hostname that resolves to a private IP (eg. try la03.int.d.sb).
You may also want to consider showing timeouts (no reply) via -O. The exact command I use on DNSTools is ping -i 0.5 -c 5 -O <ip>: https://github.com/Daniel15/dnstools/blob/master/src/DnsTools.Worker/Tools/Ping.cs#L41

Why are you using SIGKILL (process.kill(9)) rather than SIGTERM when reading stderr? https://github.com/ericls/looking-glass/blob/3bb51bdeadf562e323b36911e6b441ad3ec5fe63/src/socket.ts#L64

Didn't see your comments before for some reason. Wow, great feedback!

ericls

@FAT32 @yoursunny
Got some time to update it.
https://github.com/ericls/looking-glass/releases/tag/v0.3.0

Levi

You should put indication of action. At the moment, if my ISP do not allow PING and I perform ping test, it just shows nothing and suddenly spits out 100% loss result.

Neat script, thought, a bit overcomplicated with Deno.

ericls

@LTniger said:
You should put indication of action. At the moment, if my ISP do not allow PING and I perform ping test, it just shows nothing and suddenly spits out 100% loss result.

Neat script, thought, a bit overcomplicated with Deno.

Good point. It's not printing stderror right now...