Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Setting up High Performance SMTP to send 300,000 mails daily
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Setting up High Performance SMTP to send 300,000 mails daily

I'm working on a project for a salesforce app for a 15 years old portal with a massive user base of around 3 Million subscribers and growing. The App sends our around 10,000 to 30,000 Transactional Mails like daily Birthday mailers, personal evaluations, predictions and other notification & the company sends Mailer campaigns 2-3 times a month to whole user base and want to increase the frequency.

They have been blindly doing it from their 2 Servers, but now want an organized environment with detailed Reporting on Delivery, open and Click Tracking of transactional as well as Campaign mails.

Outsourcing this to an ESP like Mandrill & sendgrid is being considered, however the costs are exorbitant for the client right now, and they have asked me to design and setup something "self hosted". Hence I seek some education in trying out a cost effective solution for the above scenario
I have quite a good experience in Windows, and fairly good at Exim on Debian so I’m open to going either way… not biased to OS choices.

I’m also fully aware of all details for a setting up a functional mail
server, like SPF, DKIM, rDNS, Forward rDNS A record etc.

Cost of hosting a VPS (win or linux) is fairly cheap nowadays, and since the mailing server will not have any other services like Web, DB, incoming Mails, etc., I guess I’ll be good to go with a 2GB KVM SSD VPS. Please suggest!

Now the question I’m facing myself is, that all in all I will need a very efficient SMTP system to be able to send out 3 to 5 mails per second… and even more if they demand.
I’ve been reading a lot and want to know if Exim is capable of sending mails @5mails/sec so that I can achieve threshold of 15000 mails and hour. I understand that I’d need to pause and delay in between to avoid Gmail/Hotmail/Yahoo banning me for SMTP flood.

I’m a bit perplexed and Anxious as I’ve not dealt with such huge mailing requirement uptill now so please bear with my questions to follow, I certainly seek education and information in meeting this challenge efficiently.

  1. Even at 5mails/sec speed it will take me 2-3 days to cover the complete mailing list. Can I open additional mail queues within Exim/Windows SMTP to parallel send mails from 4-5 or more different mail queues?

  2. If not, can I setup 2-3-4 separate VPS instances, and distribute mailing list into respective chunks (say alphabetically ordered) and make the VPSs send mails in parallel.

I’m wondering what MTA system/software are big ESP services like Mandrill, Sendgrid, constant Contact using. I’m sure they are home baked speciality but still there must be some starting point to learn.

That said I also need recommendation what software to use to manage my mailing list, Newsletter and Sending capabilities. Most important aspect is managing bounces (both hard/Soft) and one click unsubscribe for users. Again I’m open to get PHP/python web based software or even use a windows desktop app ( I can use remote Desktop to windows VPS).

Thanks for reading up this long…
To summarise, I need suggestions to setup a high speed Mail Sending system @5mails/sec and also manage my mailing campaigns, list, unsubscribe features through a good software.

Thanked by 1mrTom
«1

Comments

  • MaouniqueMaounique Host Rep, Veteran

    These are 2 different things.
    One is a mail management system and another is the mail server itself.
    A mail server to do the actual sending and receiving is simple and will have no performance issues. I suggest using qmail as it has a lot of hooks for your other component, but, it depends on what are the requirements for the management system.
    I would start from choosing a complete management system and then selecting the backend to do the tedious work of sending and receiving.

    Take a look here if you do not wish to build your own:

    http://www.openemm.org/

    Thanked by 1mehargags
  • Of course everyone would have his own preferences for mail server software. My current preference would be postfix. Just put it on a SSD server and 5 mails / second would be nothing to worry about.

  • phpList has interesting bounce handling capabilities. Plus add-on autoresponder module, which might come in handy. You can use phpList to send the mails out, too, but I suggest setting up a separate MX for that (my preference is Postfix).

  • tommytommy Member

    exim or postfix will just fine. And you don't need SSD just because that fast.

    If you're going to send that many email what you should worry about is your IP reputation, you can't get 'green' reputation in second.

    To manage your mailing list try PowerMTA

  • Most of mailing servers has good performance, but you will have very little % of deliverability.

  • If you want to go big then use PowerMTA and interspire, powermta as a mail transfer software, interspire as a campaign management software. Beware of the people who provide installation for both of these for few hundreds bux, they use pirated software, which does not worth anything and it is illegal after all. Instead buy the licenses for both, then get yourself a decent dedicated server with at last /27 IP range. You'll have to properly setup powermta to use multiple IPs, but port25 ( the developers ) will be happy to set everything up for you ( for a small fee of course :) ) and interspire is easy to use, but you may want to get multithreading addon so you would make it much faster, but you can also tweak mysql base, that requires some knowledge on sql administration although.

    Also don't forget that you need powerful server as well to carry on all above, don't even think about the server which does not have at last 12 GB of ddr3 RAM, SSD disk ( forget about HDD with interspire ), a very good multicore CPU ( at last quad core xeon ).

    If you can't invest that much, you can do the following :

    1. Get a dedicated server with /27 ip range but with lower specs, you can have one for around $100 USD/month
    2. Install cpanel/whm on it, then get 30 domains or few domains and create multiple subdomains, each on dedicated IP, setup email account, spf and dkim of course. You can even use 15 days trial of whm, you won't need it after you configure everything afterall.
    3. Get email sending software, i can recommend you this : http://codecanyon.net/item/mailwizz-email-marketing-application/6122150 , i have it and for that price is a beast. You can host that on a separate server, you would need at last 4 gb of ram and dual core CPU. If you don't want to use web application, you can use desktop app such is http://www.atompark.com/bulk-email-sender/

    Good luck :)

    Thanked by 1KeyJey
  • henkbhenkb Member

    I have been working for a email marketing company as programmer that used their own written software. We have been using Cisco Ironports (hardware+software) at first, and we have been using PowerMTA on our own hardware.

    Both good appliances.

    But we also used to configure EXIM with rotating outgoing IP addresses, with its mailqueue in memory (outerwise the IOPS got to high for normal hardware (pre-ssd) ).

    Goodluck

  • VitaVita Member
    edited March 2015

    I heard about http://sendy.co/. I'm not sure though if they are only for newsletter. Check their website, they seem cheap. I think you can modify their application.

  • The problem isn't the MTA. Exim or Postfix could easily handle 1 Mio messages/day on a mid range VPS.

    The problem is that you want to send Millions of messages, many or even most of which risk to be considered (or actually are) spam.

    And the problem is, pardon me, that you lack the required know-how. Don't get me wrong, it's not shameful or bad to sometimes need a bit of advice or a detail. But it's a serious problem if you, as your many threads show, lack the foundations and basically don't know any of the relevant basics.
    Sorry for sounding blunt but it seems to me that you are selling a service to a client without having a reasonable basis to provide it.

    Having said that you will need a) some kind of database, b) some kind of mailer, and c) some kind of control and report interface.

    Which leads to the next problem: You will get lots of advice about mysql and exim, no doubt. But the question is whether these are adequate for you, and even more so considering your rather limited know-how and experience.

    Finally, sending, uhm, "massive mailings", is a business area in itself and I doubt that those experienced in that field will just give away for free their "magic sauce", their know-how.
    Also keep in mind that the capability to send out Mios of email per day of standard MTAs has been designed with a purpose in mind, namely many, many clients, most of them sending relatively few emails.

    Thanked by 1ricardo
  • try sendy.co . 1 buck per 10k emails. we use it.

    Thanked by 2vimalware aglodek
  • 3 Mio subscribers * 2.5 mailings/month (based on OPs spec), even ignoring birthday and other "small fries" mails -> 7.5 Mio emails/month @ 1$/10k emails -> 750$/month.

    I might be wrong but from what I read in the OP that's far beyond what they have in mind in terms of cost.
    From what I see the OP has more something in mind like 3 VPS @ 20$ each -> 60$/month plus a selfmade software solution (based on free building blocks).

  • aglodekaglodek Member
    edited March 2015

    @mehargags said: I’m also fully aware of all details for a setting up a functional mail server, like SPF, DKIM, rDNS, Forward rDNS A record etc.

    SPF, DKIM, rDNS, check. But what's this last (Forward rDNS A record) and what's that for MX-wise?

    @bsdguy:

    @mehargags said: …the company sends Mailer campaigns 2-3 times a month to whole user base and want to increase the frequency…

    So I think 10M+ emails ~$1000+/mo would be closer to the mark. And that's to start with!

  • mehargagsmehargags Member
    edited March 2015

    Thanks @dariobl
    PHPList is on my top list.. and I'm aware of the micro/macro settings needed to fine tune it as the management software.

    What I was more concerned at was if EXIM/Postfix are capable of this SMTP load, and I feel relaxed I can do it.

    Yes I've been reading through PowerMTA and Interspire at many places... and it looks like a really good solution for upto millions per day or even per hour. And Yes... special thanks to warning me about pirated installations, I was pretty confused on reading about them on some !ahem! forums, people doing it for <$200 while I roughly checked the PowerMTA license itself costs upwards of $3000... am I right ?

    Can you guide me where I can get genuine license but cheaper to over the counter rates if I must use PMTA.

    The hardware you suggested looks beasty... Im sure it is needed for PMTA+Interspire for 100K mails per hours or more, but I need a lot less.

    @Henkb
    plus I read someplaces that IP rotation can be bad for your campaings, while I sincerely feel without "paralleling" million mails per hours isn't even possible.

    @Vita Sendy is for Amazon SES, it is a little costly compared to my budget, so I'm looking to setup my own system as an experiment and exercise.

    Another options I'm checking are:
    Bitsteer BulkMTA http://www.bitsteer.com/products/bulkmta/
    and
    Greenarrow-engine and http://www.drh.net/products/greenarrow-engine/ both of which are commercial solutions... and I'm trying to get pricing from them.

    Sincere thanks on educating me over the subject. All in all... I'm trying learn my way up to setup a good system over time, cure and mature it to yield better results later, and ofcourse save BIG money down the line.

    I know ESP services are better but as the open source community notions... if it exists -use it, if it doesn't -create it! I'm trying my luck with the best intentions, I'm confident you all guyz will help me on this. That said, I'm open to pya a reasonable fee for the consultation and setup support for this too. so feel free to PM me.

  • Exim can easily handle the load, ever accidently set up an open relay?

    I can't comment on the marketing/ip rotation/reputation side of things, but don't worry about MTA even on $5 a month VPS.

  • MaouniqueMaounique Host Rep, Veteran

    linuxthefish said: Exim can easily handle the load,

    I saw some sending 1 mill an hour. We had a big mailer customer which we removed his IP from automated supervision on smtp volume but he got hacked so he went on sending millions of mails in a few hours.

    Thanked by 1linuxthefish
  • hmmm... Makes me relaxed as I know Exim better than postfix being a Debian guy.

    while the conversation is on, can you guyz recommend some good hosting providers for VPS/Dedi at low costs to setup up this mass mailing ?

    I may need good amount of IPs in the end if the mailing is more frequent, to maintain IP reputation.

  • MaouniqueMaounique Host Rep, Veteran

    mehargags said: I may need good amount of IPs in the end if the mailing is more frequent, to maintain IP reputation.

    In order to maintain IP reputation you do not need more IPs, but you need to avoid spamming, making sure all your customers want to get those emails.

    Thanked by 2vimalware Gallus
  • TrafficTraffic Member
    edited March 2015

    @Maounique said:

    There will always be someone who uses the "spam" button to unsubscribe, even if you make unsubscribing as easy as possible. Big ISPs make reporting spam easier than unsubscribing.

    Thanked by 1geekalot
  • MaouniqueMaounique Host Rep, Veteran

    Traffic said: There will always be someone who uses the "spam" button to unsubscribe

    I know, hence the making sure part.

    For example send all the transactional emails from the server and all newsletters from another service.
    Everyone wants to receive an email with password changing links, but not all wish to receive newsletters.

    Thanked by 1deadbeef
  • @Maounique said:
    Everyone wants to receive an email with password changing links, but not all wish to receive newsletters.

    Yes That's exactly the plan... transactional mail will be sent from the Webserver, all other Daily routine mails as well as Marketing mails will be sent from Separate Mailing subsystem!

    I know good list hygiene and content will keep my reputation good, but sending such enormous numbers, I read IP rotation will be required... though not to an extent fo 50-80 IPs but certainly 5-10 IPs (as told to me by some hosting providers themselves who approve bulk mailing from the dedi's)

  • geekalotgeekalot Member
    edited March 2015

    @mehargags:

    For large legitimate email campaigns, some viable commercial offerings:

    • PowerMTA (100,000+/hr per "virtual mta")
    • StrongView (500,000+/hr - but you may run into various limits on recipient email systems, etc)

    All dependent on individual message size, bandwidth available (i.e., above numbers were based on 100mbit/s pipes and other limiting factors), and number of IPs.

    (I have not worked with these systems directly, I know someone that has.)

    Thanked by 1mehargags
  • @mehargags The problem is not in speeding up Exim. Talking of that, you can emulate load balancing for Exim by using dummy SMTP listener which redirects requests more or less evenly across several Exim instances running on 127/8 (e.g. 127.10.0.1, 127.10.0.2, choose whatever you need) on the same computer.

    I have real-life experience of creating the mailing list manager you wish to create. The problem was not in speeding up the sending and using authentication. The problem was in reacting quickly on delivery problems, spam complaints etc.

    IPs take time to earn reputation of good and rules-abiding sender. It takes a single mistake to destroy that reputation.

    Unless you are ready to battle the above problems 24/7/365.25, I recommend using Amazon SES as the cheapest 3-rd party solution, talking of cost-performance. It handles the above nightmare issues itself very good.

    Thanked by 2vimalware mehargags
  • @Master_Bo

    That is part of what I meant above. The standard MTAs are designed for and act based on the assumption that legitimate email are sent.

    Sending "grey emails" is a completely different thing. There is many small issues and details a standard MTA can "rudely" step over - but a grey email engine can't. While, of course, quite some tuning and some limited add ons is feasible with a standard MTA, there is many issues to be adressed for the kind of mass mailing discussed here.

    And that's what I meant in my first posting. Someone who has very limited understanding of even everyday standard mailer setup and operation, of databases, and other technology involved (as quite some threads of the OP amply demonstrate) is left with but 1 way, namely to purchase the services or know-how.
    The best thing to do would probably be to honestly tell the client about ones strengths (which mehargags probably has, maybe in e.g. website design) and about ones weaknesses.

    Of course (not too gross) weaknesses can be compensated to a degree by asking for advice. But repeated questions that basically come down to "can you tell me how to do my job" won't help much in compensating major lacks in even the foundations. The OP occasionally talks about his will to learn. That is doubtlessly commendable and positive but one should also understand that while still learning the basics one is not in a position to sell rather advanced services in the very area where one makes ones first steps.

  • Thanked by 1netomx
  • @bsdguy said:
    Master_Bo
    Sending "grey emails" is a completely different thing. There is many small issues and details a standard MTA can "rudely" step over - but a grey email engine can't.

    Whereas I agree with you upon most you've said, I would like to comment on this.

    Multiple spam complaints and delivery problems are quite common on whitest of mailing lists. It's just human nature. Subscribers forgot they have actually subscribed and clickon "Report spam"; they send angry messages "you ***** ****, remove my email address!" instead of clicking on "Unsubscribe", receiving mail service can block your sending addresses if they caused too many bounces etc. That's just how it all goes on.

    Building a mailing list manager is a good practice in understanding the actual problems of this area, even though the experience is mostly bitter.

    Thanked by 2vimalware Maounique
  • @Master_Bo

    Misunderstanding. I was meaning technical issues.

    Let me put it this way: As a normal, say ISPs, MTA one can behave somewhat "rude" or have attitudes. After all, one is just a normal MTA sending mostly legitimate mails. With mailing lists it gets a little hairier because many receiving MTAs might get suspicious.
    When sending "grey email" (or spam) one must overcome certain issues because, unlike a normal MTA, one doesn't really have human senders; retrying, for instance, is burdensome for grey email or spam engines; same with secondary MX MTA's, etc.

    Regarding the perspective you took, you are, of course, right. No matter how honest double opt-in and what not there will always be people who will blame someone else for their problems.

  • @bsdguy said:
    With mailing lists it gets a little hairier because many receiving MTAs might get suspicious.

    This is very true. Case in point is ATT, unless you have significant reputation, they blacklist very quickly. It only takes a few suspicious e-mails.

  • I understand now the risks... and thankful for all the red and green flags mentioned by the great members. However I am determined to try this out and get my hands dirty atleast. I think the best way to measure risk and problems is to try them for real!

    I am considering **PowerMTA **purchase, and **GreenArrow **Engine (costlier than PMTA) and also **StrongView **(thanks @geekalot) which I stumbled upon yesterday.

    **
    I'm sure doing good homework, planning and learning can make you reach the peak, and I have this lovely community to help!**

    Still listening to the ideas... keep pouring!... Thanks

  • @mehargags said:
    I understand now the risks... and thankful for all the red and green flags mentioned by the great members. However I am determined to try this out and get my hands dirty atleast. I think the best way to measure risk and problems is to try them for real!each the peak, and I have this lovely community to help!**

    Just do your customers a favor, don't make them your guinea pigs.

    @mehargags said:

    I am considering **PowerMTA **purchase, and **GreenArrow **Engine (costlier than PMTA) and also **StrongView **(thanks @geekalot) which I stumbled upon yesterday.

    I think you're still focusing on the wrong thing, the specific MTA to choose is probably the least of your worries.

  • mehargagsmehargags Member
    edited March 2015

    @Microlinux said:
    guinea pigs

    No way... they know this would be an Experiment, and I already have their approval...!

    I think you're still focusing on the wrong thing, the specific MTA to choose is probably the least of your worries.

    No Again... Commercial MTA's like PMTA, Green Arrow and SV give you keen stats and analytics on bounces/deliverability and other important factors which help you fine tune your deliverability.... which ultimately is what everyone is pointing at.

    I am not as naive as the thread may suggest...

Sign In or Register to comment.