New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What's better for huge static website?
Hello there,
I'm going to make a static article site. But I'm wonder 10k-30k articles on database (like Mysql, tuning by Redis or Memcache) is better than 10k-30k articles on 10k-30k static html files.
And what about if more than 300k articles?
Is anyone experience with big site like that can help me?
Thanks in advance.
What's better for huge static website?
- What's better for huge static website?60 votes
- Database (SQL or NonSQL)16.67%
- Static html file with SSD or RAM Disk83.33%
Comments
Static html files would make your site faster; however, if ever you need some fancy functions like commentaries or logins, database would be needed.
SSD or spinning rust?
I'd go with disk. less overhead... the OS disk cache effectively works the same as MyISAM tables, but without the application logic overhead. You'll also have slightly more memory for the disk cache without MySQL running.
Static files will always be faster than any backend server with a database. However, you can use a HTTP cache like Varnish to achieve similar results. Just pick the right tool for the job.
You don't need a RAM Disk, you can just cache everything in RAM.
You are going to go nuts maintaining a huge static site like that if you don't have a server side application helping you out. Remember you can always automatically generate static pages from your dynamic pages for maximum performance. You can do fastcgi caching in nginx for example. You get the benefits of both dynamic and static pages when you do that.
Serving static files should generally be faster - just make sure you have enough RAM for the OS to cache files appropriately and you should be good to go. The OS will take care of keeping recently referenced files around evicting stale files (etc.) but a caching proxy like Varnish will also help (more so because of application/content specific logic, while the OS is at the mercy of a lot of other "demands")
Also, depending on the size of the files and how compressible they are, you should also consider storing them compressed and have them served directly without the overhead of (re)compressing them on the fly to clients (that support compression). It should also save a fair bit of IO at the expense (possibly) of a bit more CPU (which should still be better than IO waits).
Plus with appropriate expiry headers you can go a long way in terms of intermediate caches and the likes.
How about cloudflare's "cache everything"
Since it is static , CF caches it , you are done , less bamdwidth , less request to server, no more trouble , no more worries
Thanks guys. That's exactly what I worry, huge IOPS if go with disk, and although with RAM cache, I don't know it will better performance/price than database or not.
Maybe I will choice hybrid as your suggestion (I'm going to add to the poll but I can not edit it
). And with some dynamic contents like comments, like.. will go with self-hosted disqus.
By the way, CloudFlare seem NOT cache everything unless you use Enterprise.
.
Correct me if I am wrong
You can make Cloudflare cache everything via page rule even on the free plan.
Not necessarily...lots of people using static + disqus and systems like that.
OP could use a static site generator like jekyll, pelikan, hugo, or one of a dozen others. Write a new page, run generate, and it rebuilds the site (usually only content that's changed).
Static website using some control panel should create pages instantly.
https://getgrav.org/
Came across this related/relevant discussion:
https://news.ycombinator.com/item?id=14550060
I'd recommend in-memory storage (Redis) instead of static HTML files, flexible and future-proof:
Use NGINX as a proxy for express js with Redis as a database, you'll have the best of all worlds.
You can use Netlify to host static sites for free if you want to cut costs, they are pretty cheap as well and can handle large amounts of traffic as they have a huge cluster of servers handling their service.
Possible, but scaling redis beyond a certain cluster size is not as simple as it sounds (neither is MySQL/MariaDB HA wise though), aside of that if you have memory.... why not? MemSQL is however also a very good candidate for such things.
Else: static... what size do we talk here? Sure, lot's of HTML but what actual disk usage? 2 1TB SSDs for a RAID1 do not cost insanely much (or more smaller ones). HDD rarely makes sense if large random read and not much in-ram caching (eg. a lot of random requests or small RAM) but only for large files, and even that is simple solved then by rewrite/similar (or on proxy) to a HDD/SSD-cached/NAS array.
Avoiding DB and having a plan to scale easily - static files certainly are if you have a way to push updates reliable and fast to all edges, but ultimately you just end up with proxies then anyway which you could do, like, in nginx, yo - sounds more reliable and easier to maintain in most cases.
Will your entire website fit within the amount of RAM available for your server budget? If yes, then serve all the files from a ramdisk using nginx
Otherwise, will your entire website fit within 2-2.5x the amount of RAM you will have? If yes, then look into lz4, lz4hc or zstd compressed ramdisk
Otherwise, get the fastest SSD or HDD setup that you can afford (ideally 2x NVMe in RAID0). Run varnish or simply just nginx (filesystem caching is generally good enough)
Next,
1. Choose a static site generator from https://www.staticgen.com/
2. Set it up either (ideally) offsite or on the server (make sure you have backups especially if you are using RAID0 on the server), together with all your source files.
3. Run the generator
4. Rsync the output to Ramdisk or storage
5. If using Ramdisk, you will need a script to rsync from the staticgen output to the ramdisk during bootup
https://www.bludit.com/
100% static, don't use any database. It will faster, much faster than any dynamic website.
Use nginx and offload images/css/js to different server, if you had budget use CDN.
Nosql in ramdisk
I would have a Backend like WordPress where I cN curate the site and make it the way I like. Then have a scraper making a static copy of the pages into a public nginx frontend that can be syncing between nodes via rsync and then roundrobin through dns like cloudflare, use BunnyCDN to reduce load on the nodes and boom.
static is always better.If you have decent RAM, the OS will automatically keep the files in io RAM cache and use Nginx etc you can serve the files with very little overhead.
I would also recommend having a dynamic site. And generate the static version , because your articles are not going to change.
Because this way you have benefits of both. Only keeping static is not good for future may be?
I would co with static website on ssd + cloudflare to minimize css + caching for faster loading.