Ask Slashdot: Experience Handling DDoS Attacks On a Mid-Tier Site? 197
New submitter caboosesw writes "A customer of mine recently was hit by a quick and massive DDoS attack. As we were in the middle of things, we learned that there are proxy services of varying maturity to deal with these kinds of outbreaks from the small and mysterious (DOSArrest, ServerOrigin, BlackLotus, DDOSProtection, CloudFlare, etc.) to the large and mature (Prolexic, Verisign, etc.) Have you guys used any of these services? Especially on the lower price point that a small e-commerce (not pr0n or gambling) company could afford? Is a DDoS service really mandatory as Gartner now puts this type of service in the same tier as SEIM, firewalls, IPS, etc?"
Best defense: Overprovisioning and cutoffs (Score:5, Insightful)
There's two key strategies to avoid being DDoSed... first, have more processor, network speed, and disk I/O resources than you need for normal load so that the attacker can't fill one of your computers pipes. Then, host your server or servers at multi-connected datacenters which can cut off large users of your server before it reaches your NIC card. Firewalls at the server can't get back the bandwidth lost to needless connections, but firewalls at the datacenter entry points can. Basically, make sure none of your time-sensitive loads reach 100% and you're fine.
Re: (Score:3)
Basically, make sure none of your time-sensitive loads reach 100% and you're fine
As long as the people running the datacenter aren't metering the noise that hits the interface set up to catch your traffic, whether or not they pass it along to your server. If you're burning up their resources, somebody's got to pay the pipe(r).
Re:Best defense: Overprovisioning and cutoffs (Score:5, Funny)
Re: (Score:2)
So basically, surviving a DDoS is nothing more than a brute force battle of attrition where the goal is to have more resources than the other guy can take down.
Re:Best defense: Overprovisioning and cutoffs (Score:4, Interesting)
Typically, yes (assuming your OS platform of choice doesn't have some other resource that can be remotely exhausted more cheaply then bandwidth). The problem is one of the standard defender delimas: The attacker needs bandwidth for a short period of time (typically), as their goal is to make you say "Uncle" weather that means paying their ransom, capitulating to some demand or whatever. You as a defender have to incur a cost for your defensive strategy that is either (relatively) low, non-scalable, and continuing (trying to out provision the attacker) or a high cost outsourcing solution. The attacker on the other hand rents 10,000 nodes for 200$/day. Figure that's about 5gigs conservatively (we'll say .5mbit upload as an average per node). Now assuming your data center will handle a sudden 5gig burst without cutting you off (good ones will, cheap ones will just cut you off) your hosting bill just went up by 54TB (5*3600*24/8) per day. That's not going to be sustainable for long.
That's why the outsourcing solution tends to be the way to go if you're being targeted by anyone willing to spend halfway decent money on attacking you. The ROI from the attacker POV looks pretty good. Say they ransom you for 50K (an average number for such things). If they have to keep you under DDOS for even a week till you cave, (378 TB worth of data) that nets them 48600. That's a pretty good business case from their point of view.
It's one of those moments when it sucks to be the good guys.
Min
Re: (Score:2)
This is the first time I've heard of demands for ransom. I suppose it shouldn't be a surprise that's the motive, and come to think of it, it shouldn't be a surprise that corporations aren't eager to admit to having been extorted.
Re:Best defense: Overprovisioning and cutoffs (Score:4, Interesting)
In the case I was involved with it was wired via Western Union to a place in Moscow where (according to the PI we hired) it was picked up by call girls and taken back to the culprits. They did eventually get nailed but it took years due to the complexities of law enforcement in an international environment.
We eventually signed with Prolexic to stop them coming back.
Min
Re: (Score:2)
[citation]http://www.cbc.ca/news/technology/story/2006/10/28/online-gambling.html[/citation]
Re: (Score:2)
You are welcome to, but if the story I quoted before doesn't convince you feel free to search google news. I would suggest "Gambling site" "Random" and "DoS" as search criteria.
Min
Re: (Score:2)
Basically, make sure none of your time-sensitive loads reach 100% and you're fine.
We did that, and then attackers started to flood us with over 10gbit/s of traffic that just flooded the network link and basically forces us to null route the service.
Load balancing and an experienced sysadmin (Score:5, Insightful)
With that being said, hiring a third party ddos mitigator is entirely a cost benefit analysis that should be done on your end. Can whoever's providing your hosting now provision some extra servers and some harried sysadmins to keep you floating? See if you can ask for additional service support from your current provider.
Re:Load balancing and an experienced sysadmin (Score:5, Informative)
In most cases, your customers are going to exist in one or a few countries. It would be valuable ahead of time to add redirect rules to your iptables for entire ranges of IP addresses located in countries that don't host your customers. Redirect these IP ranges to a sacrificial server on a different pipe to the backbone. That way, when some of your customers are abroad and need access to your services, they can still get some amount of response.
Additionally, you can proactively parse your user accounts for IP addresses and build a whitelist ruleset for your iptables to implement in a defcon 0 situation. Don't use this as a normal operations mode, just when the shit has really hit the fan and you need to block everyone except your known-good account holders.
Seth
Re: (Score:2)
Redirect these IP ranges to a sacrificial server on a different pipe to the backbone.
More appropriately, have your upstream provider(s) null route these address blocks before they get to your uplink.
Re: (Score:2)
Re:Load balancing and an experienced sysadmin (Score:5, Insightful)
Having been the target of an HTTP-DDOS attack, I can tell you that manually blacklisting IP ranges is really ineffective. A DDOS botnet is comprised of thousands of machines that have been randomly infected by whatever vector the botnet operator used: Emails, web drive-by, etc. The result is that the source addresses are scattered widely with little relation between most participating addresses.
To defend against the attack, I wrote up an automatic firewall blacklisting program that detected and blocked each participating IP address individually in near-realtime. I was blocking more than 31,000 separate addresses before the DDOSers finally gave up trying to down the attacked website. Wierdly, there appears to have been no motive at all for the attack, yet they spent weeks attacking the target machine and actively trying to tune their attack to get past my filtering.
Re: (Score:2)
Get multiple transit links...
Advertise your range over the links to the locations where the vast majority of your customers come from, and then advertise everything else over a very slow backup link.
Most sites cater to local customers, so have 10gb local peering and maybe 10mb international transit... Chances are most of the ddos drones won't be local, and so the attack will quickly destroy the 10mb link while your customers are happily using the 10gb peering.
If there are ddos nodes local to you, the isps h
Re: (Score:2)
Could you post a link to your program's source code? Or, if you don't want to, could you give us some pseudocode?
Yeah, I'd second that
Re: (Score:3, Funny)
Re:Load balancing and an experienced sysadmin (Score:5, Interesting)
The essence comes down to two things. Neither is particularly complicated in principle, although getting it right can be a bit fiddly.
1) Detect attacking IPs.
HTTP Flood DDOS bots aren't (at least not yet) smart enough to look and behave EXACTLY like people using web browsers. They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets. They make sequential requests from the same IP address - but with different user agents. They might load a web page that uses cookies - but never return the cookies that are set. Or they might return a cookie - but from a different source address or with a different user agent. They might send user agents that haven't been in widespread use in half a decade. They might not set the 'referer' header, or some other header that a browser DOES set correctly. They probably don't follow HTTP redirects. What you are looking for is any behavior that distinguishes the good traffic and the bad traffic.
So I 'tailed' the web server log and analyzed it in one to ten minute chunks to detect abnormal accesses. All detected addresses were added to a persistent database of blacklisted addresses.
2) Add the detected attacking addresses to an efficient firewall.
A naive firewall blacklist might try to just put each addresses in one big long list. This doesn't scale well beyond a couple of hundred attacking addresses. On the older machine I had, I used a 'divide and conquer' approach: I created a few hundred filter chains based on a /n subnet division of the attacking ip addresses. I then wrote a set of rules that divided incoming traffic into those chains based on the /n they were a member of. That made the number of rules required to filter n attacking IP addresses scale as about O(log n). If I had had a more recent kernel I could have used a hashed map of addresses to take that down to O(1).
After that it became a slow game of cat and mouse. The attacker would alter his attack to try and slip by the detection, I would update the detection software to detect something else he wasn't getting perfect if he managed to by-pass the filters. After about two weeks they quit attacking the web server.
The largest issue I had really was that I was starting my defense from a 'standing start': I had to write all the needed scripts from scratch while the attack was still on going.
Re: (Score:2)
Instead of trying to "detect" a DDOS, you can simply replace 1) with rate limiting. Say you're not willing to let any single IP address hit you more than 10 times a second, for example. You can even bucket rates. Say 10 is an attack.
For 2) instead of putting everything in a firewall you can have whatever is your closest-to-the-client server throw away a request, send an empty reply, send a captcha page or whatever else you think is appropriate given the rate being exceeded.
I also think you left out an impo
Re: (Score:2)
That's a great thing for doing in the kernel. Thus, I was out searching if it was alreay done, and yes, it is [debian-adm...ration.org].
Wowsers... (Score:2)
"They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets."
Just saying-- so do the blind....
Re: (Score:2)
So *that's* where the blind societies have been getting all their gold and jet planes from! B^>
Rgds
Damon
Re: (Score:3)
This assumes they are just trying to flood the httpd with requests, because doing so requires less resources on their part, and generally only harms the target box and not the isp hosting it.
If you block an attack like this, you run the risk that the attacker will switch tactics and start simply flooding your line. If there were 31,000 attacking drones, and assuming a rather conservative 512kbit/sec upstream per box thats over 10gbit/sec of traffic hitting you...
Also when sending raw packets the source addr
Re: (Score:2)
Why do you think that it worked? Load decreased?
The goal of DoS is to make your server unusable to the clients. More likely than not, your "appliance" blocked most of legitimate requests, thus fulfilling the purpose of the attack.
Re: (Score:2)
Load balancing is a given. Even to handle normal traffic you have to distribute it to your front ends somehow. But a sysadmin blacklisting IP address ranges? Oh my. You do not want this task to be done manually. A human, or a team of them, isn't going to be able to keep up with with a distributed attack, and they're going to make mistakes when determining which addresses to block. Remember that in a distributed attack they're not all coming from the same address and there's not much reason to expect them to
Lived Through This (Score:5, Interesting)
Re:Lived Through This (Score:4, Interesting)
Re: (Score:3)
Unless the third party proxy were the extortionists DDOSing you.
Good business model!
Re: (Score:2)
What was the name of the service you used?
Re:Lived Through This (Score:5, Informative)
Re: (Score:2)
Yeah, been there. Luckily enough, the guys that tried it with us were incompetent, and we could filter their attack because of it, but always on the lookout in case the next ones that try it are a bit smarter.
Re: (Score:2)
It was a lot cheaper to pay a third party proxy ...
How do you determine if the third party proxy has sufficient bandwidth to handle the DDoS + regular traffic?
Re:Lived Through This (Score:4, Informative)
How do you determine if the third party proxy has sufficient bandwidth to handle the DDoS + regular traffic?
They have a performance guarantee, and don't get paid if they can't keep up at the promised level. Any of the ones you'll want to use will have a dashboard that shows you a more-or-less-real-time view of the blocks/passes, and how much of the purchased throughput you're using.
Re: (Score:2)
Definitely. For a mid size site, $400 a month is practically nothing. If you can make your DOS problems go away for that, do it.
Re: (Score:2)
Not very competent asshats if they didnt notice the proxy setup, and simply continue hitting the original ip ranges...
Re: (Score:2)
Gambling (Score:2)
Have any fellow Slashdot readers tried running a gambling site without such protection? Is it reasonable to assume we'll be enough under-the-radar at first to avoid the attacks?
What Are The Odds (Score:5, Insightful)
That all these "services" are part of a protection racket?
"Oh...having DDOS problems? Just sign up with our service and we can help you out."
While not as crude as burning down building, DDOS attacks are a perfect persuader to grow your business.
I figure this is half tin foil hat and half probably real, given the things organized crime has been into in the past. It's perfect actually, you don't have to hurt people, the attacks can't be traced and your "protection" can be fine tuned to avoid looking suspicious.
Re:Gambling (Score:4, Interesting)
I used to run infosec for one of the mid-tier online gaming operations run out of the Caribbean. We got extorted by one of these gangs, and ended up paying Prolexic (they were Digidefense at the time) to solve this for us.
As for weather you can risk doing without it depends strongly on what your user tolerance for downtime is and how bursty your revenue stream is. The lower the tolerance and/or the more bursty the revenue stream the more vulnerable you are to these sort of attack methodology, as the opposition pays for the time they are actually attacking you, so if you can weather the attack they'll eventually give it up. If on the other hand they can cost you significant sums of cash by taking you out for 6 hrs (say sports betting, target the opening day games), that increases your susceptibility to these attacks.
Feel free to drop me a line if you have any more questions (my /. listed email will get to me).
Min
Post-mortem: Admin investigates attack (Score:5, Informative)
Remember this really cool slashdot story about a sysadmin on the receiving end of a DDOS?
http://slashdot.org/story/01/05/31/1330202/post-mortem-of-a-dos-attack [slashdot.org]
The original writeup link is dead but I found it here (warning: PDF). This was a really cool story.
http://www.stanford.edu/class/msande91si/www-spr04/readings/week1/grcdos.pdf [stanford.edu]
Still waiting for Spoofarino ... (Score:2)
I've lost count of how many years I've been waiting for the Spoofarino utility
And I'm not alone, other people also been talking about that utility
http://www.spywarepoint.com/gibsons-spoofarino-utility-t54570.html [spywarepoint.com]
Misunderstanding (Score:5, Informative)
The mere question of how to mitigate a DDOS indicates a fundamental lack of understanding of how IP networking and DDOS works.
You (the ISP customer) have no ability to control what packets are sent to you over your uplink circuits. You can control what you send, but you have no ability to control what you receive.
Read the sentence above. Repeat as necessary.
Even if you knew with 100% certainty which packets were "bad" packets and which were "good" packets, if your uplink is saturated, dropping them on your edge router/firewall/whatever is 100% ineffective.
The best mitigating strategy is that you need to have an agreement with your ISP and plan in place prior to an attack. Identify the hostile addresses, give them to your ISP, and they will null-route those sources either within their core or even at the edges of their networks to prevent entry. Your ISP has the capacity to mitigate a DDOS attack, you as the little customer do not.
Ah, but you CAN do something (Score:5, Informative)
Even if you knew with 100% certainty which packets were "bad" packets and which were "good" packets, if your uplink is saturated, dropping them on your edge router/firewall/whatever is 100% ineffective.
Your "best strategy" advice is very good, but it is not the "only strategy."
As others have said, you can also have multiple entry points all sharing the same back end. Each of these entry points can be on their own hosting provider. In principle, you can arrange for the front-end/back-end connection at your front-end provider to NOT share a physical wire with the "public" side of your front-end, so if it gets hit hard it crowd out traffic going to/from the back end.
Here's an example:
I run poormeddosvictim.com. I have servers at 3 sites around the country, 1.666.3.4 1.2.666.4, and 1.2.3.666.
For some reason, some mining company on Mars thinks I am evil so they keep DDOSing me.
Hosting provider A is widely connected. I advertise 1.666.3.4 so all but one of A's pipes see direct connections. I use A's remaining pipe to connect back to my back-end. I work with A so the traffic to the back-end never shares a wire or router with incoming traffic. Bang on A's incoming pipes all you want, I'll still be able to talk to my back end unless you crash me entirely.
I have similar arrangements with hosting providers B and C.
I put my back end at hosting provider D and, just for grins, have a backup back end on hosting provider E that syncs up regularly with the back-end on D.
Re: (Score:2)
Depends on the type of DDOS you are receiving. If it is a distributed DDOS then you can't do much other than having multiple pipes into the system; however, if there are not too many systems launching the ddos then you can have your provider drop those packets before they are sent down the pipe, ie you can determine what gets sent to you.
For a distributed attack you can host a couple of gateways on a larger pipe that talk to a local backend. When I was handling mission critical stuff back in the day that
Re: (Score:3, Funny)
...a distributed DDOS...
Would that be a DDDOS? And what would you do if it were a distributed DDDOS? I bet you'd be totally hosed then.
Re: (Score:3)
Heh,
Good point. I guess my mind automatically equated DDOS with a DOS. I've called it DDOS so long that I didn't even think about what DDOS stood for. Thanks for the reference check.
Re: (Score:2)
Re:Misunderstanding (Score:5, Interesting)
From our experience packet flooding attacks are rare, most are application layer attacks because they're cheaper:
- If your landing page is dynamic chances are a small site can be choked at the database from a few hundred zombies, and it's much harder to detect the zombies from the genuine clients in a safe automated fashion
- If you don't have a lot of CPU at your firewall layer you can't create long enough rule tables to stop the bad traffic as you detect it
- Often you can simplify your rules but just starting by blocking China, Russia, Korea, then smaller countries that are hosting bots.
If they are genuine flood attacks:
The idea that your ISP will block a "list of addresses" is comical, it's not nearly responsive enough, and if you're lucky your ISP will agree to block countries and only if you have a business account which you're paying over the odds access fees for. Some will even null route YOUR IP instead of the attackers to save their own infrastructure: http://www.abc.net.au/4corners/content/2009/s2658405.htm [abc.net.au]
ANDREW FOWLER: The Russian cyber attack was so sustained it backed up through Telstra's network, knocking out the whole of Alice Springs, part of Adelaide, and Telstra central in Sydney.
DAN CRANE, FORMER TECHNOLOGY MANAGER, MULTIBET: And that's when they sort of started to panic a bit I think because all of a sudden it wasn't just a, you know run of the mill attack, this was a pretty hardcore attack because that's when it started, that's when it took out Alice Springs, that's when it degraded Adelaide and that's when it melted their routers in Sydney so that's when they said that's it, we don't want a bar of it.
ANDREW FOWLER: According to Dan Crane, Telstra stopped accepting any of Multibet's internet traffic from entering Australia.
Not to mention even creating this list is a continual task. Botnets rent out "so many connections", but the computers that are active at any time rotate in and out of the pool. We saw probably around 1000 computers at a time hitting the firewall, but from a pool of more like 100,000 addresses we discovered over the course of a week. We initially took a strategy of programmatically blocking individual IPs as they came in at a response rate of about 5 seconds with some scripting, but soon moved to blocking entire countries that we didn't do business with and doing some daily post processing of the IP list as well to consolidate IPs into /27s and sometimes as far as /24s
Our last client to have this issue used Black Lotus and they seemed to do a good job for the price and be quite responsive, though they were still learning their trade at that time... I don't think they were terribly cheap. And botnets are much much cheaper, so unless you're lucky and it's someone that loses interest and not a competitor attacking you it can end up making your web hosting very expensive.
Re: (Score:2)
You (the ISP customer) have no ability to control what packets are sent to you over your uplink circuits. You can control what you send, but you have no ability to control what you receive.
Yes you do have lots of potential control. It's just expensive to exercise. Because you have to have multiple datacenters and multiple uplink circuits, to have any control (without intentionally breaking connectivity).
You buy transit and peer with major providers in different datacenters and anycast your server IP a
Re:Misunderstanding (Score:5, Informative)
This is a bit of a naive explanation.
Let me explain how a DDoS mitigation strategy works for many of the companies listed in the summary. They setup datacenters in 10, 15, or more places all hosting a proxy. Some of these solutions use DNS to route traffic around problems (GSLB) while others like CloudFlare use Anycast which is awesome and super hard to get right. Each of these services are typically setup with tons of bandwidth capacity, well over 10Gb, but often times into the 100Gb range. They also often have deals with upstream providers that can filter traffic at the edges meaning it never makes it onto the internet in the first place.
Since you servers are not exposed to the internet, and the ones are are have far, far more horsepower to deal with this than a DDoS will even manage from the client side they can easily just churn through the attack, discarding connections and never letting them hit your limited servers. This is how they can easily survive Anonymous style DDoS attacks.
The other thing is to ensure you have turned of every "feature" your load balancer is giving you. SSL termination at the LB, full session management, etc. All of these cost load balancer CPU which is easy to take advantage of, even if there is a DDoS mitigation system in front of your site. You can't just add a few more servers either. Adding capacity to a load balancer is nearly impossible to do mid-attack.
Even more interesting is that you can often times trick the crappy ddos software by doing things like excessively slow responses (tarpitting) making its loop take ages to try again. This is pretty much using the tactics of a DDoS directly against the attackers.
Another common tactic is to add attackers to a view in your bind config that resolves your hostname to 127.0.0.1 just for them. This works if you do not have long TTLs and they are using hostnames. If they are using direct IPs then you simply move your traffic to a second IP and drop the one they are attacking. Best case is if you can do this via BGP announcements so the traffic simply will fail to route and everybody wins.
And yes, I do this professionally but not for any commercial product.
Re: (Score:3)
Outsource it (Score:3)
I've lived through this (although in my case the twits doing it were holding us for ransom) Prolexic was the solution we went with and I endorse it. The economics of the situation strongly favor outsourcing to a third party. It's a service you'll likely need for a short period of time, provisioning it yourself would entail obtaining equipment and specialized expertise that you would have to commit to over a long period of time. A Prolexic can afford to obtain better equipment, and have specialized staff who can configure it to block the latest attack because they're dealing with it for clients constantly.
Min
First Hand Experience at Small Company (Score:4, Interesting)
Posting AC as I would prefer not to expose my employer in anyway.
I went through this exact situation the week before last Thanksgiving last year. I work for a gifting retailer that makes all of its money in Q4. Not a good situation. We're a small - mid sized business with about $20 million in sales from our Ecommerce site.
We went the cheap route first. The proxy service cost about $500/month and guaranteed 10 Mbps clean traffic to the site. Our DNS was changed swinging our domains to the proxy service and ACLs put in place on the "backend" to only allow connections from our new gateway in the proxy.
Things were fine for about 24 hours when the attack was stepped up. The service was seeing 450 Mbps inbound to our main domain. That is not a mistake - 450 Mbps is easily attained using a botnet or simply focusing the attention of some lurkers on pastebin links. We now had to change DNS AGAIN to "upgrade" to their better platform that could handle this attack. As we started this work, we also began talking to a couple of the higher end services...
After the $500/month service capped out and blew a gasket, we made the tough decision to go with the Cadillac. It was costly and they had us over a barrel (day before Thanksgiving, cheap service not working out, "sure would hate to see your site go down on Black Friday" mob pressure). But we knew even half a day of lost demand would pay for the yearly service (yes, it is yearly - no month to month option).
The difference was amazing. As soon as we had swung our DNS over to the new guys, the attack was mitigated within 5 minutes and abated within 20. This, of course, leads the paranoid to wonder whether it was the service doing the attacking to begin with, but we are a high profile target in the minds of the Occupy movement, so it made sense (I do not share my employers sense of community - it is only a job).
We have been attacked since then and every time the attack was mitigated within 5 minutes. If you require this type of uptime, build this service into your budget from the beginning.
Re: (Score:2)
I guess it's expensive to be unpopular.
Re: (Score:3)
I guess it's expensive to be unpopular.
No, it's expensive to be hated by a small group of people who happen to not mind being network abusing dicks so they can put on a show of hurting your business. There are plenty of people who don't like the Occupy nonsense, but most of them are too busy doing something constructive to sit around figuring out how to DDoS attack George Soro's hundreds of web properties.
Unfortunate name (Score:2)
No matter how many times I see it, my first thought on seeing "DOS attack" is that a virus downloaded MS-DOS onto a computer.
Which would almost be the worst thing to happen to a computer.
Re: (Score:2)
Maybe it was: a prophylactic is a preventative measure. Like taking vitamins to avoid colds or locking your door to prevent theft. Prophylaxis need not involve genitalia, latex, and smirking. B^>
Rgds
Damon
Akamai is a good alternative (Score:2)
MaxCDN & CloudFlare (Score:2)
Post a link on Slashdot! (Score:2)
I want to see this website!
Don't ask on /. (Score:5, Insightful)
This is a discussion you need to take to the NANOG list. Don't ask the amateurs, ask the professionals. The answer will involve ACLs, BGP settings, and community strings. If you don't have your own ASN then you need to push the issue upstream and work with your provider. Period. If you do have your own ASN and are running BGP then you need to read the NANOG list (and learn to take shit from Randy Bush, et al. They know what they are talking about.) Asking on /. can only make things worse.
easy (Score:2)
2) change username/password of superuser account from 'god'
3) profit, since Gibsons easily survive ddos attacks/flooding as evidenced by the documentary released over a decade ago detailing attempts to hack a Gibson
a mirror (Score:2)
Update mirrors with a script.
The ideas (prejudices) of monarchy, monotheism, etc. are set firmly in our heads, but why your e-commerce should have only one and only URL? Let it have 2 or 3 URLs.
We use a mirror for the 4th year, and it is the 4th year of sanity for me.
Pay your protection money (Score:3)
To most of the commenters: WTF? You have obviously never been involved in a DDOS attack. Here is why:
1) A typical DDOS attack in 2012 will send traffic measured in hundreds of MBPS/GBPS down your pipe. Not only is this a massive volume of traffic, but almost all of it is in the form of SYN/ACK packets (which are exponentially more difficult for your frontend servers to handle; especially when they are never followed by a FIN.) This is many orders of magnitude more difficult to deal with than what most sites are scoped for. You cannot just "handle it," we're talking about something that is often 7-8 standard deviations away from your "normal" peak traffic levels. In other words, your infrastructure cannot handle it. Because if you overbuilt your infrastructure to those levels, you are an idiot. DDoS protection services cost a fraction of what it would cost you to build a network that could handle that. /b/tards decided to DDoS us, but I took care of it 3 months ago."
2) Your normal DDOS doesn't come from one "large user." (hence, the first D in DDoS.) It comes from thousands (or hundreds of thousands) of IP addresses, all at once. Botnets? Yeah, they are real things, and they can be really destructive. And bad people control them, and you may have fired their mother at one point. Who knows why they have it in for you, but they probably will at some point.
3) Even if your infrastructure could handle an amount of legitimate traffic equal to the volume the DDoS will produce over the span of 6-12 hours, you would then have to pay for it. I promise you, you don't want to be in that position. Most hosting providers probably won't make you pay for all of it, but they will become real interested in what you're hosting that would make someone want to DDoS you in the first place. And your boss will probably make you find a proxy solution to solve the problem; so why not be proactive about it so you can say "Yeah, those
TL;DR: DDoS proxy services like CloudFlare exist for a reason: it's simply not economically feasible to overbuild your infrastructure to the point where you could survive such an attack. Pay the man, keep your site up, and ignore the punks smashing cars in the street because you have insurance, so fuck em.
Redirect everything to /. (Score:2)
Obvious solution. You will have invented the reverse slashdot effect [wikipedia.org] .
Ignore gartner... (Score:2)
Don't pay any heed to what the likes of gartner say, none of these things are "mandatory", they are all a factor of risk vs benefit vs cost...
If your running a webserver, it only has tcp/80 open and nothing else... Then you add a firewall which sits infront and only allows 80/tcp, what have you gained? You may have an extra point of monitoring, and mitigation against outbound traffic *if* the box gets compromised... But if someone exploits a vulnerability on port 80 it's not going to help you (and there was
Had good results using blockdos.net (Score:2)
If your hosting service doesn't have an Anti-DDoS tier or option available, the people at blockdos.net were able to help in a pinch. If your host can change your IP address, you'll get the best results. You point your domain at the BlockDoS provided IP and then set your firewall (or server if firewall not an option) to block any inbound traffic not coming from a second BlockDoS provided IP. The downside is that you lose a lot of request header information and your server logs show all requests coming fro
Re:Change Apache to nginx (Score:4, Informative)
And yes, I DO use nginx, and it rocks. It's just not the silver bullet you're talking about.
Re:Change Apache to nginx (Score:5, Informative)
2) Configure the firewall to proxy TCP hand-shakes, so your web servers don't get flooded with syn packets unless the hand-shake actually finishes
3) Mid-grade nginx web server will handle 70k+ requests/sec
4) Setup your DNS to round-robin to several web servers
Between your firewall and your webserver rules, you should be able to filter most obvious DDOS's. That which you can't filter, you'll just have to brute-force it and suck it up.
Your web servers can handle more requests than you have bandwidth, the next bottle-neck is your database.
There is not "silver bullet" like you said, but a properly designed system should be robust enough to leave your bandwidth your bottle-neck Most web apps I see aren't designed to properly make use of SQL. It's like someone trying to shoe-horn procedural logic into a database. Gotta get your DB architect to work with the programmers.
A properly architected web app with a properly architected DB should be able to handle more requests than your bandwidth can handle.
The only real DDOS to worry about is a flood and you can't really stop that unless it's a simple up-stream change. Enough machines DDOS'n ping floods at you will take you down. Filter all you want at your router, you won't have the bandwidth. Would be too simple to filter up-stream. A bunch of random forged TCP packets will suck up your bandwidth. If the attack is well distributed, ain't not'n you can do about it.
There is not "Silver Bullet" like you said, but a properly designed system should have bandwidth as its bottle-neck
Re: (Score:2)
You could server your site from multiple EC2 regions, and use anycasting if you don't need to manage state server-side. You're going to pay out the nose for traffic and instance time, but Amazon's AWS should have *more* that enough capacity to server as a sink for all but the largest of attackers.
Re: (Score:3)
Re: (Score:2)
Ignore this, since it wouldn't help all that much. I'd say that https://www.varnish-cache.org/ [varnish-cache.org] can help, but to be honest, if you want to be up, just stick them on Amazon Cloud services or something. They'll have a really hard time getting to that, and you'll leave all the defending and whatnot to be someone elses' job
Re:Change Apache to nginx (Score:4, Interesting)
Amazon AWS bills you bandwidth directly. A DDoS could get very expensive.
Re: (Score:3, Informative)
Re: (Score:2)
this is not true anymore, amazon does not charge for inbound traffic.
G-Wan ! (Score:3)
Instead of nginx, I'd use g-wan
It's very efficient, even more than nginx
Re: (Score:2)
Because you love writing web sites in C?
Re: (Score:2)
No, because it's fast
Besides, I figure there's no harm to utilize my knowledge in C to make my site super efficient
Re: (Score:2)
Nginx is not really going to help if you are getting DDOSed and your incoming pipes are full.
The problem isn't at the web server layer, it's at the network layer.
Re: (Score:3, Insightful)
If it helps against DDOS attacks, how is it stupid advice?
Because it doesn't really and you're just being a fanboi?
Re: (Score:3)
Re: (Score:2)
Re:Change Apache to nginx (Score:4, Informative)
It doesn't help against DDoS attacks. Not even remotely, not even a little bit. To put the advice to a metaphor, a DDoS attack is where there are so many people loitering in the front lobby of a business that people can't even get into the front door of a building. Using a different web server is like having a receptionist who speaks faster; it doesn't address the nature of the attack in the slightest way possible. These attacks are either driven by saturation of network links or by leveraging vulnerabilities in underlying database-driven applications (hint: a little-known SQL command called WAITFOR is often to blame); using nginx won't help in the slightest bit.
Christ...these attacks are over a decade old; read up or be quiet.
Re:Change Apache to nginx (Score:5, Informative)
My take on this is that nginx is cool for static pages, we all should know that.... new optimisations in Apache 2.4 hope to address some of these and Apache is easier for me to configure for dynamic sites with controllers.
Regarding DDOS - neither of these will help... there are different types of DDOS attacks, sure. Any site that is dynamic in nature is screwed by any DDOS before it even saturates the entrance because an inability to disseminate requests in time causes the webserver to effectively stall. There are mitigations, one of the best is iptables rate limit for DOS attacks, of course defending DDOS attacks requires enough horsepower behind the scenes, so that when the entrance is saturated, requests can still be distributed usually by a load-balancer that places the bottleneck at the entrance alone - placing the site in the cloud with auto-scaling will solve this at a cost. Any type of DDOS attack that relies on an exploit though, requires a fix, removal or workaround before any horsepower mitigation can take place.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
> In fact most DDOS attacks rely on causing heavy load on the server.
If that's the case you are far worse off using nginx. 'Untuned' apache will limit your poor webserver to 256 running threads. With defaults nginx will start 1000 processes, which makes your site four times as dead. And if you are clueless but know how to google you can easily get at 10.000.
The kind of attack that relies on high load preys on clueless admins who think that 'tuning a webserver' is maxing out all limits.
Re: (Score:2)
Re: (Score:3)
Apache can be configured to drop sessions more quickly to get past the trauma, but the hosts need more configuration. Shortening TCP session duration is key, but so also is going to something else than BIND, which is also less survivable than other DNS servers, IMHO. There are some reasonable TOE card, router, and layer 2/3 switch configs that can also help cut down the pain.
Load balancing helps, watching syslogs for weird behavior and using a syslog manager to alert you when various events occur, all these
Re: (Score:2)
Re: (Score:2)
I agree. It seems like your hosting was using the attack as an excuse to try shoving an upsell down your throat.
Any transaction that allows someone else to take advantage of your own misfortune should be viewed skeptically.
Re: (Score:2)
The question is..
Did the attacker throw 3gbps at it because that's all they had, or because that's all that was needed to do the job?
It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port, a 3gbps attack will take all of those offline even if it doesn't harm the colo provider a a whole.
Also the isp has to pay for the bandwidth usage that hits them, even if most of it never actually reaches the end customer... A
Re: (Score:2)
It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port
Yes, you are right, but if you have DDNS worries, you'll keep away from those providers, and/or opt for a beefier connection. Many of them will happily provide you with detailed information regarding network structure, so you can be aware of what you are actually paying. Also, many of those "cheaper" providers have somewhat frequent DoS attacks inside their own network (some machines get compromised, and then try to scan/infect everybody else), so, again, handling/filtering a 3Gbps attack isn't usually a bi
Re: (Score:2)
To put it in p
Re: (Score:2)
If you don't have the infrastructure and/or the staff, outsourcing DDOS protection seems a good choice. Trying to protect against the 3Gbps mark is a bit silly, since next time you may get hit by 10x that traffic. It is important to learn the nature of the attack (are we talking about hundreds of ip addresses, or a dozen? are they "residential connections" or compromised servers? is it a TCP handshake attack, or requests to specific pages? do th
Re: (Score:2)
Annoyingly we haven't had an attack since
What a coincidence eh?
Re: (Score:2)
You are partly correct but you are oversimplifying things.
You assume that in a DDOS attack, your upstream capacity is 1) over-saturated and 2) the only thing that is over-saturated, or at least that nothing else would be saturated if your upstream capacity were bigger.
If your DDOS attack is not saturating your upstream, either a) you are successfully fending it off, or b) you are still suffering but increasing your upstream capacity is not the answer.
In the case of b), the suggestions you call irrelevant ar
Re: (Score:2)
Question: How to protect against DDoS attacks without regard to the availability of the target of the attack Answer: If your carrier supports it, initiate an RTBH on the targeted IP. If not, contact the helpdesk and have them null route it.
NO. That is not a DDos protection strategy. That is a "white flag" strategy. It is essentially surrender, because you are intentionally making the attack succeed against the target then.
Re: (Score:2)