[{"data":1,"prerenderedAt":2179},["ShallowReactive",2],{"nav-products":3,"articles-list":10},[4],{"name":5,"tagline":6,"to":7,"isInternal":8,"icon":9},"Valdras Gate","no VPN licenses to manage","\u002Fproducts\u002Fgate\u002F",true,"\u002Fpublic\u002Fimg\u002Fproducts\u002Fgate\u002Fgate_logo.png",[11,150,441,588,1358,1450,1603,1688,1799,2045],{"id":12,"title":13,"author":14,"body":15,"category":138,"cover":139,"date":140,"description":141,"extension":142,"featured":143,"isFeatured":8,"meta":144,"navigation":8,"path":145,"preview":146,"seo":147,"stem":148,"__hash__":149},"articles\u002Farticles\u002Fhow_to\u002F2020-12-19-how-to-lower-aws-costs.md","How to lower AWS Costs","David Gatti",{"type":16,"value":17,"toc":130},"minimark",[18,27,32,35,39,42,45,48,59,64,67,70,73,82,86,89,92,95,99,102,106,109,113,116,120,123,127],[19,20,21,22,26],"p",{},"This article is meant for CEOs, CTOs, and Managers. The goal is to give a better understanding of how to lower AWS costs by answering a common raised question by clients: ",[23,24,25],"em",{},"What is the simplest way to lower the AWS bill","?",[28,29,31],"h1",{"id":30},"the-mysticism","The mysticism",[19,33,34],{},"AWS costs are wrapped in mysticism due to mysterious practices by companies that specialize in this field. The reality is simpler than you might think, otherwise these companies would not hide behind big words and convoluted explanations.",[28,36,38],{"id":37},"the-aws-billing-page","The AWS Billing Page",[19,40,41],{},"AWS provides you with a very simple to use billing page, which brakes down all your costs based on the service, and the region. This makes it easy to find the biggest expense in the account.",[19,43,44],{},"More importantly if you have an organization account and under that you have dozens of other AWS accounts, the billing page also provides you with a brake down based on the individual account under the organization. This makes it simple to find the account that is generating the biggest costs, while allowing you to then drill down to the service, and region.",[19,46,47],{},"The billing page is the most important thing to use when it comes to getting an idea of what is going on in the account. Once you know where the money goes, we can go into the details and figure out if there is something that can be changed to lower the monthly bill.",[19,49,50,51,58],{},"By the way: if you want to ",[52,53,57],"a",{"href":54,"rel":55},"https:\u002F\u002Fconsulting.0x4447.com\u002Farticles\u002Fknowledge\u002Fthe-benefits-of-aws-organizations.html",[56],"nofollow","learn more about the benefits of AWS Organizations",", check out the article.",[60,61,63],"h2",{"id":62},"pay-by-the-hour-services","Pay-By-The-Hour Services",[19,65,66],{},"The majority of the costs in your organization are going to come from EC2, RDS and similar Pay-By-The-Hour type services. This are the most used services, and most importantly AWS charges you for these services the moment they are turned on, regardless of the load.",[19,68,69],{},"The majority of the time, these services are turned on by employees temporarily just to perform a test, or run a one time job, but will forget to turn them off. Services could be left on for years, while your bank account is being drained for servers that have 0.45% CPU time.",[19,71,72],{},"Thankfully this problem can not only can be quickly identified, but also be quickly solved by just simply turning off those unused servers. The simplest approach is to stop all the servers with less then 1% CPU time in the case of Linux, and 3% in the case of Windows instances. This approach allows you to stop paying for the servers, but also turn them back on when they are actually needed (Because if they are, someone somewhere should start screaming).",[19,74,75,76,81],{},"On the other hand, if your servers are needed, then you are probably not taking advantage of the Reserved Instances features that AWS provides. With this service, instead of paying by the hour, you can pay up to 3 years upfront, and thus ",[52,77,80],{"href":78,"rel":79},"https:\u002F\u002Fconsulting.0x4447.com\u002Farticles\u002Fhow_to\u002Fhow-to-lower-ec2-costs.html",[56],"lowering your bill by 40% to 75%",", check the article to find out more.",[60,83,85],{"id":84},"serverless-services","Serverless Services",[19,87,88],{},"The next big contributors are all of the serverless technologies that AWS offers. In contrast to the pay-by-the-hour onset, this one generates a charge when actually used. This means that there might not be a quick fix. To reduce the price for these services, you’ll have to change how you use them. In the case of S3 it might be as simple as enabling Intelligent-Tiering on a S3 bucket, but in other cases you might have to change the code of the application, for example how DynamoDB is being used.",[19,90,91],{},"Since we are discussing DynamoDB already - when you use this NonSQL database in On-Demand mode, you get charged by how many queries to it you make, and also how much data is being moved back and fort. If the application using this services makes lots of poorly constructed requests, or the data is structured in a way where the data transfer is high, then you you’ll pay incredible amount of money on a daily basis.",[19,93,94],{},"In situations similar to these, you’d need an expert to look in to the problem, and come up with a solution, and then this solution will have to be implemented correctly to lower the bill.",[28,96,98],{"id":97},"prevention-mechanisms","Prevention Mechanisms",[19,100,101],{},"Overall the best approach is to always educate people to help them make better decisions when they use all that AWS has to offer. There also some simple ways to remind them of what they have learned.",[60,103,105],{"id":104},"for-pay-by-the-hour-services","For:  Pay-By-The-Hour Services",[19,107,108],{},"The IAM services has the idea of Policies, this are documents that specify what a IAM User can do in the AWS account. It can be as simple as the user has only access to EC2 Instances, or you can extend it to include which type of instances can be launched by the user, and even specify the region in which they can do that. With a policy like this you can be confident that nobody will launch expensive instances without permission. This is true for any service that is pay-by-the-hour.",[60,110,112],{"id":111},"for-serverless-services","For:  Serverless Services",[19,114,115],{},"Sadly, there is no way to have a policy that prevents developers from making queries the wrong way, or storing data the wrong way. The best solution here is to train the team so they can learn all the best practices for a given AWS Service.",[28,117,119],{"id":118},"good-monitoring","Good Monitoring",[19,121,122],{},"Lastly, a big part of keeping a close eye on the AWS expenses is good monitoring. Each service that is being used should have CloudWatch Alarms monitoring thresholds, and sending notifications when there is something that goes over your set limits. Of course in some cases this takes some effort to get the threshold just right, but once you do, you’ll know exactly when and what is causing a problem.",[28,124,126],{"id":125},"to-sum-it-up","To Sum it Up",[19,128,129],{},"Depending on your situation, I hope that you’ll be able to (even today!) lower your AWS bill substantially. If not, you should now have a good understanding on what to look for, and what to ask the people in your organization.",{"title":131,"searchDepth":132,"depth":132,"links":133},"",2,[134,135,136,137],{"id":62,"depth":132,"text":63},{"id":84,"depth":132,"text":85},{"id":104,"depth":132,"text":105},{"id":111,"depth":132,"text":112},"how_to","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-aws-costs\u002Fcover.jpg","2020-12-19","The reality is simpler than you might think.","md","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-aws-costs\u002Ffeatured.jpg",{},"\u002Farticles\u002Fhow_to\u002F2020-12-19-how-to-lower-aws-costs","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-aws-costs\u002Fpreview.jpg",{"title":13,"description":141},"articles\u002Fhow_to\u002F2020-12-19-how-to-lower-aws-costs","Ue0NtBlwRuOJiKw-z6IjKYQzo5f_yZp9XtFJX9IxQ9M",{"id":151,"title":152,"author":14,"body":153,"category":138,"cover":430,"date":431,"description":432,"extension":142,"featured":433,"isFeatured":434,"meta":435,"navigation":8,"path":436,"preview":437,"seo":438,"stem":439,"__hash__":440},"articles\u002Farticles\u002Fhow_to\u002F2020-10-17-how-to-hire-the-technical-team.md","How to hire the technical team",{"type":16,"value":154,"toc":424},[155,161,165,168,171,174,177,181,185,188,192,195,198,202,205,209,212,215,218,221,224,227,230,233,237,306,310,313,321,324,328,354,358,400,404,414,417],[19,156,157,158,26],{},"This article is meant for CEOs, CTOs, and Managers. The goal is to give a better understanding of how to hire technical people by answering a common raised by clients: ",[23,159,160],{},"how can I hire technical people when I myself know nothing about the technology world",[28,162,164],{"id":163},"understanding-the-symptoms","Understanding the symptoms",[19,166,167],{},"The biggest difficulty when hiring technical people is when you don’t fully understand the skill you are hiring for and you can only try to gage their expertise with your gut feeling. You are stuck with your fingers crossed that the new hire will be as he presents himself.",[19,169,170],{},"This is an issue that very often comes up with clients that I work with. Normally, the symptoms look like this: the project starts fairly well, you see progress, and milestones are being achieved; but, after a few months, the project starts to slow down, seemingly simple things start to take months, and\u002For the new features are unstable.",[19,172,173],{},"At this point, even as a non-technical person, you start to realize that something is not right.",[19,175,176],{},"This type of scenarios is not due to the lack of knowledge of how to code (coding is easy) - the problem is in the lack of understanding that organized code is more important than simply knowing how to use a “if” statement. Or the DevOps people don't fully understand AWS, nor the best practices that they should follow to manage the whole infrastructure correctly, since AWS is not a SaaS solution like Heroku or others. AWS is the building blocks that allow you to build your own SaaS application. This key difference requires having a different mindset that not many know they have to get into, and so they manage AWS by hand, in one environment, and with minimal automation.",[28,178,180],{"id":179},"how-to-fix-the-existing-damage","How to fix the existing damage?",[60,182,184],{"id":183},"code","Code",[19,186,187],{},"When I come in to help, I do not try to understand the code, since normally the state of the code is in poor condition. There are no comments, no spaces, no new lines, and no formatting at all. I focus on reformatting the code to make it clean and simple to understand, then I do a second pass where I start to simplify some expressions, and after this stage all the mistakes that were impossible to figure out become obvious and stand out on their own.",[60,189,191],{"id":190},"infrastructure","Infrastructure",[19,193,194],{},"In this situation, we need to go over the idea of the project from scratch, understand what you want to build, and then start to make a diagram of the whole infrastructure based on what I think needs to be done. Then with the team we go over it to see if I missed something, where the team may have already encountered some limitation of the approach worth not repeating. We then finalize the diagram.",[19,196,197],{},"Once that is completed, I start making multiple CloudFormation files that describe the stack on AWS. This means that what was done will be scrapped and everything will be redone using CloudFormation files that will be deployed in three new environments: Development, Demo and Production - to keep everything organized.",[28,199,201],{"id":200},"how-to-start-with-the-right-team","How to start with the right team",[19,203,204],{},"If you have not yet start to put together a team, this section is for you. Here I want to give you a good frame of reference on how to look for technical people, while having little-to-no technical skills yourself. I believe that asking very technical questions does not allow you to find the best talent. For example, a law company won't ask a candidate if they memorized all the law books; they want to find out what type of person is in front of them. Just because you memorized paragraph 8 doesn't mean you are a good lawyer. The same analogy can be used in the technology world. Just because you know a \"if()\" statement or a function, doesn't make you a good developer - you just know how to code, and that is just a skill.",[60,206,208],{"id":207},"what-to-focus-on","What to focus on",[19,210,211],{},"You can’t ask direct questions where the answer is a Yes or No, because you will always get back what you want to hear. You have to give open ended questions that require the other side to think.",[19,213,214],{},"You want to look for people that can explain what they know and are able to share their experience easily, where you don’t have to pull information out of them. If you have the most genius person that doesn't know how to talk to others, and most importantly doesn't know how to explain technical things to non-technical people, then you'd get someone with very limited use. This person can only function with another technical person. In the beginning of your company, if they have to report to you, then you don't want someone like this on your team.",[19,216,217],{},"Another important thing is not to ask for cliché things like their education or years of experience, because this information is meaningless. One person can be very good after 2 years of learning the skill, while others will be mediocre even if they have 8 or 20 years of experience. If someone worked for 20 years in a big company, they can easily hide their mediocrity within the organization. And worst of all, they might not even know they are mediocre. Education is the same: you could have passed with the lowest grades and still got a diploma. If someone slept thought the whole university, they might know less then a passionate programmer who learned his craft in his free time.",[19,219,220],{},"Also, just because someone has been in the market for 20 years already, dose not mean they are good at it. You can infer that they just know the syntax of the specific language they claim to know; whereas the most important skills are the abilities to write clearly and to easily understand code, which are things that nobody teaches anyone. People that write hard to understand code will make your project eventually grind to a halt, and you will be held hostage by one programmer because he is the only one that knows what he did. Under no circumstance would you want to be held hostage by unnecessary complexity.",[19,222,223],{},"Writing working code is not a skill; it is a trait. The skill is to make it presentable to others.",[19,225,226],{},"Similarly with AWS. I have interviewd countless AWS “experts” who claim to know all that there is to know about AWS, yet when asked something more serious, they fail. As of writing this article, AWS has 170 services - just because an \"expert\" claimed to have used 3 of them doesn't mean they know how to use AWS. I have talked with a lot of candidates that claim they know AWS because they made and account and went over a simple tutorial on how to setup a Lambda. To them, this is the basic requirement to claim to know AWS.",[19,228,229],{},"A true AWS expert is someone that knows the ideology of the service, is able to mix and match all of this services to build a cost effective and extensible solution that can grow over time with the business, understands the pitfalls, knows how to get out of problems, and can figure out a new service within a week to combine it with the project if necessary. Not to mention they need to know how to automate everything in AWS, because nothing can be done by hand, as clicking things around with a mouse through the AWS console is something that can’t scale over time.",[19,231,232],{},"An expert also needs to understand what security is, how to provide only the required access to employees, and how to adhere to best practices to make it easier to pass compliance tests.",[28,234,236],{"id":235},"general-questions-for-all","General questions for all",[238,239,240,251,259,266,274,282,290,298],"ul",{},[241,242,243,247,250],"li",{},[244,245,246],"strong",{},"How do you keep your team leader up to date with the progress of the work?",[248,249],"br",{},"This is personal, so if you like the answer, all good.",[241,252,253,256,258],{},[244,254,255],{},"What do you do when you don’t know something?",[248,257],{},"What you want to hear is that they check the documentation, search Google, ask friends, then look for help with you.",[241,260,261,264,250],{},[244,262,263],{},"When there is nothing to do, what do you do?",[248,265],{},[241,267,268,271,273],{},[244,269,270],{},"Do you make your bed in the morning?",[248,272],{},"You want to hear a \"Yes\", this should tell you that they are more organized than most and don't give up easily.",[241,275,276,279,281],{},[244,277,278],{},"After how many years will you be done learning?",[248,280],{},"You want to hear \"never.\" This is the only acceptable answer and approach to life.",[241,283,284,287,289],{},[244,285,286],{},"Why did you get into the technology business?",[248,288],{},"If someone is in it just for the money, it might mean they don't have a mindset where they want to learn and get better.",[241,291,292,295,297],{},[244,293,294],{},"When someone tells you it can't be done, what do you do?",[248,296],{},"The answer should be: \"I don't accept no for an answer.\"",[241,299,300,303,305],{},[244,301,302],{},"How many times do you try before asking for help?",[248,304],{},"You want someone that knows after a while he needs to ask for help.",[60,307,309],{"id":308},"how-to-check-for-comprehension","How to check for comprehension",[19,311,312],{},"Write a few paragraphs describing 5 tasks to be completed but not broken down into sections or a list. Then tell them to brake down your description into steps and technologies that need to be used to accomplish the task. What they come up with should:",[238,314,315,318],{},[241,316,317],{},"Have all 5 things that need to be done.",[241,319,320],{},"Should be easy to understand by you. If something is unclear, ask for clarification and see if they can simplify the step further.",[19,322,323],{},"Unclear explanations by people only means that either they don’t understand the technology or the problem itself. Either is a serious red flag, because people will create very complex solutions that will be hard to extend over time.",[28,325,327],{"id":326},"specific-questions-for-developer","Specific questions for Developer",[238,329,330,338,346],{},[241,331,332,335,337],{},[244,333,334],{},"Do you have some open source projects on GitHub?",[248,336],{},"They should have something. This shows that they are passionate about programming and love it so much that they spend their free time learning new things on their own.",[241,339,340,343,345],{},[244,341,342],{},"Paste 100 lines of the code that you are most proud of.",[248,344],{},"Regardless of if you know how to code or not, you should look for clear comments that explain the story of the code. And the code itself should look visually simple. The more the code feels good to you, the better.",[241,347,348,351,353],{},[244,349,350],{},"What is more important, to use all the latest features of a language, or to keep the code as simple as possible?",[248,352],{},"You want to hear that they want to keep the code as simple as possible. Using the latest features of a language means that they will lose themselves in semantics, theories, ideologies, and ideas instead of focusing on the work that needs to be done. These types of people can be good scholars, but in a company when works needs to be done right, they will slow the project down and create endless discussions that go nowhere.",[28,355,357],{"id":356},"specific-questions-for-aws-experts","Specific questions for AWS Experts",[238,359,360,368,376,384,392],{},[241,361,362,365,367],{},[244,363,364],{},"How would you host a website with no servers on AWS that is SEO friendly?",[248,366],{},"You want to hear that the site will be built statically, hosted on S3, and delivered over CloudFront; while setting up a CodePipeline to build the site automatically when there is new code in the repository to build the static version of the site through CodeBuild.",[241,369,370,373,375],{},[244,371,372],{},"How many services have you used in your latest project?",[248,374],{},"You want to hear services that you think you'd like to use in your project. It is impossible to know them all, but you want to hear at least 2\u002F3 of what you need.",[241,377,378,381,383],{},[244,379,380],{},"How did you integrate them with each other?",[248,382],{},"This is a hard one because you can make a project hundreds of ways that will all work. But AWS is designed around the idea of Triggers, especially when you deal with the Serverless part of AWS. So you want to hear that they used triggers to pass messages to another service, which then triggered another one etc.",[241,385,386,389,391],{},[244,387,388],{},"How do you keep up to date with AWS?",[248,390],{},"You want to hear that they work on small private projects, check the AWS Blog daily, and watch videos regarding how others use AWS, in order to get ideas that can be implemented in another project.",[241,393,394,397,399],{},[244,395,396],{},"When would you use Server and when Serverless?",[248,398],{},"Any project that you start on AWS, you want to start with Serverless. This is an ideal approach because in the beginning you don't know how the project will pan out, and once it picks up steam, you want to convert the most used lambdas or other services in to a regular server to improve performance and reduce costs. This is the ideal road map.",[28,401,403],{"id":402},"unacceptable-excuses","Unacceptable excuses",[238,405,406],{},[241,407,408,411,413],{},[244,409,410],{},"\"I can’t share my code\" or \"I have signed an NDA.\"",[248,412],{},"A passionate developer should be proud to have one or 2 projects open for anyone to see.",[28,415,416],{"id":125},"To sum it up",[19,418,419,420,423],{},"You don't understand technical people because you lack the knowledge they have, or you don't understand them because they can't explain their work in simpler terms. The technicality is their expertise and knowledge they acquired over time, which is something that you don't have to have. But they must be able to not make you feel stupid or question your intelligence. Knowledge is ",[23,421,422],{},"not"," intelligence.",{"title":131,"searchDepth":132,"depth":132,"links":425},[426,427,428,429],{"id":183,"depth":132,"text":184},{"id":190,"depth":132,"text":191},{"id":207,"depth":132,"text":208},{"id":308,"depth":132,"text":309},"\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-hire-the-technical-team\u002Fcover.jpg","2020-10-17","how can I hire technical people when I myself know nothing about the technology world?","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-hire-the-technical-team\u002Ffeatured.jpg",false,{},"\u002Farticles\u002Fhow_to\u002F2020-10-17-how-to-hire-the-technical-team","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-hire-the-technical-team\u002Fpreview.jpg",{"title":152,"description":432},"articles\u002Fhow_to\u002F2020-10-17-how-to-hire-the-technical-team","h9c6vuD_yc_rjW34IauQptvB6Rivhw8tM05U1PLDUtY",{"id":442,"title":443,"author":14,"body":444,"category":577,"cover":578,"date":579,"description":580,"extension":142,"featured":581,"isFeatured":434,"meta":582,"navigation":8,"path":583,"preview":584,"seo":585,"stem":586,"__hash__":587},"articles\u002Farticles\u002Fexperience\u002F2020-09-06-the-best-serverless-database-combo-on-aws.md","The best serverless database combo on AWS",{"type":16,"value":445,"toc":565},[446,449,453,456,459,462,465,468,472,475,478,482,485,488,492,495,499,502,505,509,512,515,519,528,537,540,543,546,550,553,557,560,562],[19,447,448],{},"This article is meant for anyone who wants to better understand Serverless databases to get the best performance at the right price. This article asnwers an easier question: What is the best serverless database combo on AWS?",[60,450,452],{"id":451},"the-most-unconventional-solution","The most unconventional solution",[19,454,455],{},"Over the years, I came to realize that your typical SQL database that we've been using since the 80's are only good for one thing: creating reports and calculating stats.",[19,457,458],{},"SQL was typically used for everything, but nowadays we can be more specific with how we store data.",[19,460,461],{},"We now have at our disposal blazing fast databases that can handle insane amounts of read and write, that automatically scale based on traffic, while the downside is that they are bad at generating reports and summarizing data stored within them. We also have graph databases that allow us to track all the inner connections between objects, and we have databases that are very cheap, but are slow at retrieving the data.",[19,463,464],{},"Nowadays, you can pick multiple designs and store different types of data based on the use case, and this way you get the benefit of both worlds. You pay only for what you need and can manipulate the data with ease thanks to the specificity of the database.",[19,466,467],{},"Here's a simple example.",[60,469,471],{"id":470},"the-s3-and-dynamodb-combo","The S3 and DynamoDB combo",[19,473,474],{},"Based on my experience, the best database combo that also happens to be my current default approach, which is S3 and DynamoDB.",[19,476,477],{},"As far as I can see, this approach covers 80% of web applications out there. Where DDB would be used to host the data related to the application UI to render the interface as fast as possible, S3 is used to display the extra details when the user would be more comfortable waiting the extra time for additional information.",[60,479,481],{"id":480},"a-simple-example-invoices","A simple example - invoices",[19,483,484],{},"Every website with a payment option will have at the least monthly invoices generated in their system. When an invoice is generated, you can store the ID, Number, and Timestamp of the invoice in DynamoDB, and you can store the rest of the invoice details in S3.",[19,486,487],{},"This way you can display a table that loads all the rows blazing fast, which allows the user to have a quicker experience when going back to look for the right invoice. Once the user clicks on the invoice they want, you can pull all the rest of the information from S3, which is slower but acceptable by the user.",[60,489,491],{"id":490},"the-benefits","The benefits",[19,493,494],{},"This approach allows you to have a speedy UI while keeping the costs as low as possible. Costs can remain low because moving data back and forth in DynamoDB is expensive, but very cheap in S3. By combing the two, you can have the best of both worlds.",[60,496,498],{"id":497},"ideal-solution-to-test-ideas","Ideal solution to test ideas",[19,500,501],{},"This approach is ideal for startups to test an idea for a product and pay only for the resources used, because DynamoDBN and S3 are Serverless services provided by AWS where you pay only when those services are used. This means that while developing your costs will be minimal, and you will start paying more and more once you gain more and more users. Hopefully if your pricing structure for your product is correct, then you should be able to keep up with the AWS bill regardless of growth.",[19,503,504],{},"Once your business is established, you can look at the AWS bill and see if you would benefit from a transition to a different database, or if you should change how your application works to be even more efficient. The aforementioned approach is a solid start.",[60,506,508],{"id":507},"something-extra-s3-supports-sql","Something extra: S3 supports SQL",[19,510,511],{},"One important fact about S3 that not many people know is that you can run SQL queries on your data if the data is structured the right way. This allows you to store big JOSN files in S3 and query only for the data that you need. You will be able to minimize costs even more by only asking for the right information instead of pulling the whole file, not to mention that you will only need to download a fraction of the file itself.",[19,513,514],{},"S3's support of SQL is feature worth exploring and taking the time to understand.",[60,516,518],{"id":517},"real-life-example","Real life example",[19,520,521],{},[522,523],"img",{"alt":524,"className":525,"src":527},"example",[526],"uncontain","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-best-serverless-database-combo-on-aws\u002Fs3_ddb_example.png",[19,529,530,531,536],{},"The above screenshot is an example of how to organize your data between DynamoDB (in blue) and S3 (in red) using ",[52,532,535],{"href":533,"rel":534},"http:\u002F\u002Famazon.com",[56],"Amazon.com"," as a real life example. Of course this is just an idea to show how the approach explained above could be implemented, and it does not mean this is how you should always do it, or how Amazon should do it.",[19,538,539],{},"But lets drill down into the example: Lets assume you have written your code in a way that makes a call to DynamoDB and S3 in an asynchronous way.",[19,541,542],{},"First of all, I'd order my promises in a way where I'd first make all the S3 queries since they will be the slowest, and then move on to DynamoDB. Data from DDB will come sooner than S3, and so we can start displaying the page based on this information. By the time we are done, we should have the response from S3 which we can use to show the rest of the page.",[19,544,545],{},"Based on your applications and needs you can play with the order of the queries to fine tune the load time of the data.",[60,547,549],{"id":548},"what-else","What else",[19,551,552],{},"To maximize speed, especially with a website that has lots of data to display, you can consider splitting the data in to more then one file in S3. In the example above, you can have the basic description in one file, and you can have the extra description that is hidden under a page scroll in a separate file. That way you could download the second file after the first one in order to significantly speed up the page loading time, and the majority of the users won't notice that the detailed description did not show up immediately.",[60,554,556],{"id":555},"super-simple-backups","Super simple backups",[19,558,559],{},"This solution also makes it very easy to create backups. When it comes to DynamoDB, you only have to enable one switch to enable point in time recovery, while with S3 you can enable object versioning and keep, for example, 7 days worth of version for a file, thus allowing you to restore a deleted or modified file with ease.",[60,561,416],{"id":125},[19,563,564],{},"Of course over time, if your idea takes off, you could move crucial data to a faster Database to maximize revenue, but this approach would for sure help you build your idea, pay only for what you use, and not worry about database scalability and performance, since AWS guarantees the performances of these two services.",{"title":131,"searchDepth":132,"depth":132,"links":566},[567,568,569,570,571,572,573,574,575,576],{"id":451,"depth":132,"text":452},{"id":470,"depth":132,"text":471},{"id":480,"depth":132,"text":481},{"id":490,"depth":132,"text":491},{"id":497,"depth":132,"text":498},{"id":507,"depth":132,"text":508},{"id":517,"depth":132,"text":518},{"id":548,"depth":132,"text":549},{"id":555,"depth":132,"text":556},{"id":125,"depth":132,"text":416},"experience","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-best-serverless-database-combo-on-aws\u002Fcover.jpg","2020-09-06","What is the best serverless database combo on AWS?","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-best-serverless-database-combo-on-aws\u002Ffeatured.jpg",{},"\u002Farticles\u002Fexperience\u002F2020-09-06-the-best-serverless-database-combo-on-aws","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-best-serverless-database-combo-on-aws\u002Fpreview.jpg",{"title":443,"description":580},"articles\u002Fexperience\u002F2020-09-06-the-best-serverless-database-combo-on-aws","4m9KU2T1IZu4hWDZlX6TIKnERCbKKT3t4KCdBWvmUfo",{"id":589,"title":590,"author":14,"body":591,"category":138,"cover":1348,"date":1349,"description":1350,"extension":142,"featured":1351,"isFeatured":434,"meta":1352,"navigation":8,"path":1353,"preview":1354,"seo":1355,"stem":1356,"__hash__":1357},"articles\u002Farticles\u002Fhow_to\u002F2020-08-11-how-to-understand-dynamodb.md","How to understand DynamoDB",{"type":16,"value":592,"toc":1334},[593,596,600,603,607,610,614,617,621,624,636,639,643,646,651,654,783,786,789,981,984,995,998,1005,1009,1012,1178,1181,1184,1188,1191,1197,1200,1217,1235,1238,1241,1258,1261,1278,1281,1292,1295,1298,1302,1310,1314,1325,1327,1330],[19,594,595],{},"This article is meant for anyone who wants to better understand DynamoDB after experiencing an AWS bill for DDB is too high, the complexity is too big, or the promised speed is not there. This article answers a straightforward question: Am I using DynamoDB the right way?",[60,597,599],{"id":598},"why-im-not-getting-dynamodb","Why I'm not getting DynamoDB",[19,601,602],{},"You may be surprised, but 20+ years of SQL may not help you with this new AWS service, and the biggest reason is that you are probably comparing DynamoDB to MongoDB. Both are NoSQL but still are very different from one another. For example, just because you store data as JSON and there are no columns for a fixed structure that you have to set upfront doesn't mean they are the same.",[60,604,606],{"id":605},"mongodb-is-self-hosted-and-this-is-a-problem","MongoDB is self-hosted, and this is a problem",[19,608,609],{},"Because you can host MongoDB, it makes it hard for you to realize the incorrectness in the ongoing data structure. Doing it wrong doesn't appear to cost you anything because you don't have limits, and you pay just for the server that MongoDB runs on. If the performance is an issue, you can always add a bigger server, and all the mistakes get hidden away.",[60,611,613],{"id":612},"dynamodb-is-the-real-nosql-database","DynamoDB is the real NoSQL Database",[19,615,616],{},"Since AWS hosts DynamoDB for you, you are forced to adhere to the NoSQL principles, which makes you pay for all your mistakes out of your pocket, and the only way around it is to understand NoSQL.",[60,618,620],{"id":619},"dynamodb-key-limits","DynamoDB Key Limits",[19,622,623],{},"DynamoDB has three significant limits:",[625,626,627,630,633],"ol",{},[241,628,629],{},"Pay per request",[241,631,632],{},"Pay for each byte that goes in and out",[241,634,635],{},"One record can't be bigger than 400Kb",[19,637,638],{},"These three rules make it so that you either structure the data the right way or lose the performance and could have to pay thousands of dollars for simple requests.",[60,640,642],{"id":641},"examples","Examples",[19,644,645],{},"I'll give you two examples that should give you a good idea of how to start thinking about DDB data structure and organize your website to make sure that all work well together.",[647,648,650],"h3",{"id":649},"user-data","User Data",[19,652,653],{},"A typical case of user data would be the name, email, address, and a JSON object that could look like this:",[655,656,660],"pre",{"className":657,"code":658,"language":659,"meta":131,"style":131},"language-JSON shiki shiki-themes github-light github-dark","{\n    \"name\": \"0x4447\",\n    \"email\": \"hello@0x4447.email\",\n    \"address\": {\n        \"street\": \"10 Springflied St.\",\n        \"city\": \"Springflied\",\n        \"state\": \"Springflied\",\n        \"country\": \"USA\",\n        \"postal\": \"01100\"\n    }\n}\n\n","JSON",[183,661,662,671,687,700,709,722,735,747,760,771,777],{"__ignoreMap":131},[663,664,667],"span",{"class":665,"line":666},"line",1,[663,668,670],{"class":669},"sVt8B","{\n",[663,672,673,677,680,684],{"class":665,"line":132},[663,674,676],{"class":675},"sj4cs","    \"name\"",[663,678,679],{"class":669},": ",[663,681,683],{"class":682},"sZZnC","\"0x4447\"",[663,685,686],{"class":669},",\n",[663,688,690,693,695,698],{"class":665,"line":689},3,[663,691,692],{"class":675},"    \"email\"",[663,694,679],{"class":669},[663,696,697],{"class":682},"\"hello@0x4447.email\"",[663,699,686],{"class":669},[663,701,703,706],{"class":665,"line":702},4,[663,704,705],{"class":675},"    \"address\"",[663,707,708],{"class":669},": {\n",[663,710,712,715,717,720],{"class":665,"line":711},5,[663,713,714],{"class":675},"        \"street\"",[663,716,679],{"class":669},[663,718,719],{"class":682},"\"10 Springflied St.\"",[663,721,686],{"class":669},[663,723,725,728,730,733],{"class":665,"line":724},6,[663,726,727],{"class":675},"        \"city\"",[663,729,679],{"class":669},[663,731,732],{"class":682},"\"Springflied\"",[663,734,686],{"class":669},[663,736,738,741,743,745],{"class":665,"line":737},7,[663,739,740],{"class":675},"        \"state\"",[663,742,679],{"class":669},[663,744,732],{"class":682},[663,746,686],{"class":669},[663,748,750,753,755,758],{"class":665,"line":749},8,[663,751,752],{"class":675},"        \"country\"",[663,754,679],{"class":669},[663,756,757],{"class":682},"\"USA\"",[663,759,686],{"class":669},[663,761,763,766,768],{"class":665,"line":762},9,[663,764,765],{"class":675},"        \"postal\"",[663,767,679],{"class":669},[663,769,770],{"class":682},"\"01100\"\n",[663,772,774],{"class":665,"line":773},10,[663,775,776],{"class":669},"    }\n",[663,778,780],{"class":665,"line":779},11,[663,781,782],{"class":669},"}\n",[19,784,785],{},"The gut instinct is to save this whole object into DDB and call it a day. But this is not the right way of doing it. The reason: you pay per bytes sent and received. In a typical situation, you will display the user's name more than the email and then display the address from time to time. But if you save everything, you'd pay for the whole payload each time, even though you only want to get the name. This approach is slower since you have to send more data, and you end up paying for the whole payload, despite just needing the name.",[19,787,788],{},"A better way of saving this data is to split it into three different objects:",[655,790,792],{"className":657,"code":791,"language":659,"meta":131,"style":131},"{\n    \"id\": \"001\",\n    \"type\": \"user#basic\",\n    \"name\": \"0x4447\"\n}\n\n{\n    \"id\": \"001\",\n    \"type\": \"user#basic#email\",\n    \"email\": \"hello@0x4447.email\"\n}\n\n{\n    \"id\": \"001\",\n    \"type\": \"user#basic#address\",\n    \"address\": {\n        \"street\": \"10 Springflied St.\",\n        \"city\": \"Springflied\",\n        \"state\": \"Springflied\",\n        \"country\": \"USA\",\n        \"postal\": \"01100\"\n    }\n}\n\n",[183,793,794,798,810,822,831,835,840,844,854,865,874,878,883,888,899,911,918,929,940,951,962,971,976],{"__ignoreMap":131},[663,795,796],{"class":665,"line":666},[663,797,670],{"class":669},[663,799,800,803,805,808],{"class":665,"line":132},[663,801,802],{"class":675},"    \"id\"",[663,804,679],{"class":669},[663,806,807],{"class":682},"\"001\"",[663,809,686],{"class":669},[663,811,812,815,817,820],{"class":665,"line":689},[663,813,814],{"class":675},"    \"type\"",[663,816,679],{"class":669},[663,818,819],{"class":682},"\"user#basic\"",[663,821,686],{"class":669},[663,823,824,826,828],{"class":665,"line":702},[663,825,676],{"class":675},[663,827,679],{"class":669},[663,829,830],{"class":682},"\"0x4447\"\n",[663,832,833],{"class":665,"line":711},[663,834,782],{"class":669},[663,836,837],{"class":665,"line":724},[663,838,839],{"emptyLinePlaceholder":8},"\n",[663,841,842],{"class":665,"line":737},[663,843,670],{"class":669},[663,845,846,848,850,852],{"class":665,"line":749},[663,847,802],{"class":675},[663,849,679],{"class":669},[663,851,807],{"class":682},[663,853,686],{"class":669},[663,855,856,858,860,863],{"class":665,"line":762},[663,857,814],{"class":675},[663,859,679],{"class":669},[663,861,862],{"class":682},"\"user#basic#email\"",[663,864,686],{"class":669},[663,866,867,869,871],{"class":665,"line":773},[663,868,692],{"class":675},[663,870,679],{"class":669},[663,872,873],{"class":682},"\"hello@0x4447.email\"\n",[663,875,876],{"class":665,"line":779},[663,877,782],{"class":669},[663,879,881],{"class":665,"line":880},12,[663,882,839],{"emptyLinePlaceholder":8},[663,884,886],{"class":665,"line":885},13,[663,887,670],{"class":669},[663,889,891,893,895,897],{"class":665,"line":890},14,[663,892,802],{"class":675},[663,894,679],{"class":669},[663,896,807],{"class":682},[663,898,686],{"class":669},[663,900,902,904,906,909],{"class":665,"line":901},15,[663,903,814],{"class":675},[663,905,679],{"class":669},[663,907,908],{"class":682},"\"user#basic#address\"",[663,910,686],{"class":669},[663,912,914,916],{"class":665,"line":913},16,[663,915,705],{"class":675},[663,917,708],{"class":669},[663,919,921,923,925,927],{"class":665,"line":920},17,[663,922,714],{"class":675},[663,924,679],{"class":669},[663,926,719],{"class":682},[663,928,686],{"class":669},[663,930,932,934,936,938],{"class":665,"line":931},18,[663,933,727],{"class":675},[663,935,679],{"class":669},[663,937,732],{"class":682},[663,939,686],{"class":669},[663,941,943,945,947,949],{"class":665,"line":942},19,[663,944,740],{"class":675},[663,946,679],{"class":669},[663,948,732],{"class":682},[663,950,686],{"class":669},[663,952,954,956,958,960],{"class":665,"line":953},20,[663,955,752],{"class":675},[663,957,679],{"class":669},[663,959,757],{"class":682},[663,961,686],{"class":669},[663,963,965,967,969],{"class":665,"line":964},21,[663,966,765],{"class":675},[663,968,679],{"class":669},[663,970,770],{"class":682},[663,972,974],{"class":665,"line":973},22,[663,975,776],{"class":669},[663,977,979],{"class":665,"line":978},23,[663,980,782],{"class":669},[19,982,983],{},"This way, you get only what you need because you have three separate pages for each piece of information. For example:",[238,985,986,989,992],{},[241,987,988],{},"User Identity",[241,990,991],{},"Email",[241,993,994],{},"Address",[19,996,997],{},"When the user visits this individual page, you will get only what the page requires.",[19,999,1000,1001,1004],{},"Later in this article, I will write more about the ",[183,1002,1003],{},"type"," key.",[60,1006,1008],{"id":1007},"invoice-data","Invoice Data",[19,1010,1011],{},"The following example looks at the 400Kb limit per object. The clients I work with always end up with this data structure: they use the Array type object, add Invoice data in this array, and then save the whole thing as one object in DynamoDB. For example:",[655,1013,1015],{"className":657,"code":1014,"language":659,"meta":131,"style":131},"{\n    \"id\": \"001\",\n    \"type\": \"user#invoices\",\n    \"invocie\": [\n        {\n            \"id\": \"001\",\n            \"price\": \"$99\",\n            \"payed\": true\n        },\n        {\n            \"id\": \"002\",\n            \"price\": \"$99\",\n            \"payed\": true\n        },\n        {\n            \"id\": \"003\",\n            \"price\": \"$99\",\n            \"payed\": false\n        }\n    ]\n}\n\n",[183,1016,1017,1021,1031,1042,1050,1055,1066,1078,1088,1093,1097,1108,1118,1126,1130,1134,1145,1155,1164,1169,1174],{"__ignoreMap":131},[663,1018,1019],{"class":665,"line":666},[663,1020,670],{"class":669},[663,1022,1023,1025,1027,1029],{"class":665,"line":132},[663,1024,802],{"class":675},[663,1026,679],{"class":669},[663,1028,807],{"class":682},[663,1030,686],{"class":669},[663,1032,1033,1035,1037,1040],{"class":665,"line":689},[663,1034,814],{"class":675},[663,1036,679],{"class":669},[663,1038,1039],{"class":682},"\"user#invoices\"",[663,1041,686],{"class":669},[663,1043,1044,1047],{"class":665,"line":702},[663,1045,1046],{"class":675},"    \"invocie\"",[663,1048,1049],{"class":669},": [\n",[663,1051,1052],{"class":665,"line":711},[663,1053,1054],{"class":669},"        {\n",[663,1056,1057,1060,1062,1064],{"class":665,"line":724},[663,1058,1059],{"class":675},"            \"id\"",[663,1061,679],{"class":669},[663,1063,807],{"class":682},[663,1065,686],{"class":669},[663,1067,1068,1071,1073,1076],{"class":665,"line":737},[663,1069,1070],{"class":675},"            \"price\"",[663,1072,679],{"class":669},[663,1074,1075],{"class":682},"\"$99\"",[663,1077,686],{"class":669},[663,1079,1080,1083,1085],{"class":665,"line":749},[663,1081,1082],{"class":675},"            \"payed\"",[663,1084,679],{"class":669},[663,1086,1087],{"class":675},"true\n",[663,1089,1090],{"class":665,"line":762},[663,1091,1092],{"class":669},"        },\n",[663,1094,1095],{"class":665,"line":773},[663,1096,1054],{"class":669},[663,1098,1099,1101,1103,1106],{"class":665,"line":779},[663,1100,1059],{"class":675},[663,1102,679],{"class":669},[663,1104,1105],{"class":682},"\"002\"",[663,1107,686],{"class":669},[663,1109,1110,1112,1114,1116],{"class":665,"line":880},[663,1111,1070],{"class":675},[663,1113,679],{"class":669},[663,1115,1075],{"class":682},[663,1117,686],{"class":669},[663,1119,1120,1122,1124],{"class":665,"line":885},[663,1121,1082],{"class":675},[663,1123,679],{"class":669},[663,1125,1087],{"class":675},[663,1127,1128],{"class":665,"line":890},[663,1129,1092],{"class":669},[663,1131,1132],{"class":665,"line":901},[663,1133,1054],{"class":669},[663,1135,1136,1138,1140,1143],{"class":665,"line":913},[663,1137,1059],{"class":675},[663,1139,679],{"class":669},[663,1141,1142],{"class":682},"\"003\"",[663,1144,686],{"class":669},[663,1146,1147,1149,1151,1153],{"class":665,"line":920},[663,1148,1070],{"class":675},[663,1150,679],{"class":669},[663,1152,1075],{"class":682},[663,1154,686],{"class":669},[663,1156,1157,1159,1161],{"class":665,"line":931},[663,1158,1082],{"class":675},[663,1160,679],{"class":669},[663,1162,1163],{"class":675},"false\n",[663,1165,1166],{"class":665,"line":942},[663,1167,1168],{"class":669},"        }\n",[663,1170,1171],{"class":665,"line":953},[663,1172,1173],{"class":669},"    ]\n",[663,1175,1176],{"class":665,"line":964},[663,1177,782],{"class":669},[19,1179,1180],{},"User data from the example above is somewhat limited. In the invoice example, we will hopefully increase the object size each month for the product's lifespan. But at some point, you will go over the 400Kb limit per object. What then? Developers often come up with crazy solutions to split the object to go around this limit. Unfortunately, doing so adds a considerable amount of complexity. The performance also drops drastically since you have to get back 400Kb of data every time, plus you are not taking advantage of partitioning.",[19,1182,1183],{},"There is a more straightforward solution that will allow you to never have to think about this limit ever again and add incredible flexibility to your data which will dramatically increase the performance and speed.",[60,1185,1187],{"id":1186},"always-add-a-search-key","Always add a Search Key",[19,1189,1190],{},"The Search Key is your friend, and you should always use it. There are edge cases where you don't have to use it, but these are edge cases, and over time, you'll know when not to use it. Right now, we will focus on 99.99% of cases.",[19,1192,1193,1194],{},"The search key needs to be understood as a file path, the same file path on your computer. For example: ",[183,1195,1196],{},"\u002Ffolder_name\u002Ffolder_name\u002Ffile_name.extension",[19,1198,1199],{},"You have to create the search key using the same logic. For example, if we take the user data example, we could have:",[238,1201,1202,1207,1212],{},[241,1203,1204],{},[183,1205,1206],{},"user#basic",[241,1208,1209],{},[183,1210,1211],{},"user#basic#email",[241,1213,1214],{},[183,1215,1216],{},"user#basic#address",[19,1218,1219,1220,1223,1224,1227,1228,1231,1232,1234],{},"With a structure like this, you can use DynamoDB's query function and the filter ",[183,1221,1222],{},"start_with",". With these two things, you can get everything that you need in one go by stating that you want everything that starts with ",[183,1225,1226],{},"user#basic#"," or, to be more precise, with a regular ",[183,1229,1230],{},"get"," request, in order to only get ",[183,1233,1216],{}," for the page that just shows the address.",[19,1236,1237],{},"You still have the flexibility to get a big chunk of information when needed, but you also get all the granularity you need most of the time.",[19,1239,1240],{},"The same rule applies to the invoices because you can have a search key like this:",[238,1242,1243,1248,1253],{},[241,1244,1245],{},[183,1246,1247],{},"user#invoice#nr1",[241,1249,1250],{},[183,1251,1252],{},"user#invoice#nr2",[241,1254,1255],{},[183,1256,1257],{},"user#invoice#nr3",[19,1259,1260],{},"And you can keep all your data in separate records, where 400Kb is plenty. But you also can go one step further and do this:",[238,1262,1263,1268,1273],{},[241,1264,1265],{},[183,1266,1267],{},"user#invoice#2020#nr1",[241,1269,1270],{},[183,1271,1272],{},"user#invoice#2020#nr2",[241,1274,1275],{},[183,1276,1277],{},"user#invoice#2020#nr3",[19,1279,1280],{},"This way, you can make a query that returns:",[238,1282,1283,1286,1289],{},[241,1284,1285],{},"just one invoice",[241,1287,1288],{},"the whole year's worth of invoices",[241,1290,1291],{},"or everything together.",[19,1293,1294],{},"As you can see, this is your limitless array of data that can go into infinity. No need to worry about the 400Kb limit, not to mention that working with arrays in DynamoDB is complex and wasteful.",[19,1296,1297],{},"To update an entry in the array, you need to know the position of that item (index), which means that if you don't know it, then you have to download the whole object, loop over every entry in the array, find the position, update it, and then send back the whole object again. So if you have an object of 400Kb, you then have to send back an additional 400Kb, which is 800Kb in total. So, again, a slow and expensive operation to change one item in an array.",[60,1299,1301],{"id":1300},"never-do-this-with-dynamodb","Never do this with DynamoDB",[238,1303,1304,1307],{},[241,1305,1306],{},"Never make one big object",[241,1308,1309],{},"Never use Arrays unless the data you are working with has them, and you need to store it as is.",[60,1311,1313],{"id":1312},"always-do-this-with-dynamodb","Always do this with DynamoDB",[238,1315,1316,1319,1322],{},[241,1317,1318],{},"Brake down your data in the smallest parts you can",[241,1320,1321],{},"Design your search key as folder structure",[241,1323,1324],{},"Always make precise queries to get what you need.",[60,1326,416],{"id":125},[19,1328,1329],{},"I hope this explanation will give you a much better understanding of how to go about working with DynamoDB, and I genuinely hope this will bring your costs down and speed improvements.",[1331,1332,1333],"style",{},"html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":131,"searchDepth":132,"depth":132,"links":1335},[1336,1337,1338,1339,1340,1343,1344,1345,1346,1347],{"id":598,"depth":132,"text":599},{"id":605,"depth":132,"text":606},{"id":612,"depth":132,"text":613},{"id":619,"depth":132,"text":620},{"id":641,"depth":132,"text":642,"children":1341},[1342],{"id":649,"depth":689,"text":650},{"id":1007,"depth":132,"text":1008},{"id":1186,"depth":132,"text":1187},{"id":1300,"depth":132,"text":1301},{"id":1312,"depth":132,"text":1313},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-understand-dynamodb\u002Fcover.jpg","2020-08-11","Am I doing DynamoDB the right way?","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-understand-dynamodb\u002Ffeatured.jpg",{},"\u002Farticles\u002Fhow_to\u002F2020-08-11-how-to-understand-dynamodb","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-understand-dynamodb\u002Fpreview.jpg",{"title":590,"description":1350},"articles\u002Fhow_to\u002F2020-08-11-how-to-understand-dynamodb","rAkIITqguB5LCqpL2CB5JpcuWL9DhJWtVMpbP6MqJWg",{"id":1359,"title":1360,"author":14,"body":1361,"category":138,"cover":1441,"date":1442,"description":1368,"extension":142,"featured":1443,"isFeatured":434,"meta":1444,"navigation":8,"path":1445,"preview":1446,"seo":1447,"stem":1448,"__hash__":1449},"articles\u002Farticles\u002Fhow_to\u002F2020-07-17-how-to-organize-a-project-with-cloudformation.md","How to organize a project with CloudFormation",{"type":16,"value":1362,"toc":1434},[1363,1369,1373,1376,1379,1383,1386,1390,1393,1413,1416,1419,1423,1426,1429,1431],[19,1364,1365,1366],{},"This article is meant for CEOs, CTOs, and Managers. The goal is to give you a better understanding of how to organize a project with CloudFormation, by answering a common question clients have: ",[23,1367,1368],{},"How should a project be broken down to keep it flexible?",[60,1370,1372],{"id":1371},"dont-paint-yourself-in-to-a-corner","Don't paint yourself in to a corner",[19,1374,1375],{},"CloudFormation files are powerful, will simplify your life, and are easy to be reused in other projects. In this article where I describe the beefiest of CloudFormation files.",[19,1377,1378],{},"It is important to understand how to separate the files in to smaller stacks. Putting your whole project in to one big file will make your life harder, not to mention there is a limit of 200 services per file.",[60,1380,1382],{"id":1381},"harder-how","Harder how?",[19,1384,1385],{},"AWS will follow the instructions in the CloudFormation file and will add, update, or remove services at your command. But sometimes you can encounter a situation where you can't create an update because there is a dependency that is blocking the resources that you want to update. A common problem is, for example, renaming a service or removing a service that another service is relying on. And, since the two series are linked together by an ARN in some situations, AWS won't update that ARN and you will be stuck.",[60,1387,1389],{"id":1388},"the-solution","The solution",[19,1391,1392],{},"The key is to keep the CloudFormation files separate for each category of your project. For example, this is a default breakdown that I do for clients:",[238,1394,1395,1398,1401,1404,1407,1410],{},[241,1396,1397],{},"Database",[241,1399,1400],{},"Compute",[241,1402,1403],{},"Cedentials",[241,1405,1406],{},"SMTP",[241,1408,1409],{},"Notification",[241,1411,1412],{},"etc.",[19,1414,1415],{},"In the Database CloudFormation file, I might have RDS, DynamoDB, and also S3. If I'm taking advantage of the triggering system that DynamoDB and S3 provides, then I paste the ARN of the Lambdas from the Compute CloudFormation file. Since it is impossible right now deploy a stack that has the triggering service and the triggered one, since, for example, S3 depends on the existence of a Lambda to get the ARN to be triggered, and the Lambda permission depends on the S3 existing to get the ARN for the policy.",[19,1417,1418],{},"The solution is to keep these two parts separate to allow yourself maximum flexibility while also keeping things organized in a logical way.",[60,1420,1422],{"id":1421},"easier-to-manage-and-control","Easier to manage and control",[19,1424,1425],{},"This way of braking down the project into separate CF files makes it also easier to manage, because you can have an expert for each project. For example, if your project gets big enough, you can have a Database expert that manages and has only access to the Database stack. Same for a Serverless expert, and maybe one person that can handle 3 smaller stacks at the same time.",[19,1427,1428],{},"This way everyone only has access to what they are responsible for.",[60,1430,416],{"id":125},[19,1432,1433],{},"I hope this article helped you to get an idea how a project should be structured with multiple CloudFormation files, and of course you are not limited to the examples provided above. You are free to brake your project down in anyway that makes sense to you, but regardless of how you do, one big file is never the way to go, since the goal is to have the project grow over time.",{"title":131,"searchDepth":132,"depth":132,"links":1435},[1436,1437,1438,1439,1440],{"id":1371,"depth":132,"text":1372},{"id":1381,"depth":132,"text":1382},{"id":1388,"depth":132,"text":1389},{"id":1421,"depth":132,"text":1422},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-organize-a-project-with-cloudformation\u002Fcover.jpg","2020-07-17","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-organize-a-project-with-cloudformation\u002Ffeatured.jpg",{},"\u002Farticles\u002Fhow_to\u002F2020-07-17-how-to-organize-a-project-with-cloudformation","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-organize-a-project-with-cloudformation\u002Fpreview.jpg",{"title":1360,"description":1368},"articles\u002Fhow_to\u002F2020-07-17-how-to-organize-a-project-with-cloudformation","1Mkls91SIKHkqCVRI8oVdgXU5JyK9b-47SBn_-WpYL0",{"id":1451,"title":1452,"author":14,"body":1453,"category":1593,"cover":1594,"date":1442,"description":1595,"extension":142,"featured":1596,"isFeatured":434,"meta":1597,"navigation":8,"path":1598,"preview":1599,"seo":1600,"stem":1601,"__hash__":1602},"articles\u002Farticles\u002Fknowledge\u002F2020-07-17-cloudformation-tips-and-tricks.md","CloudFormation Tips and Tricks",{"type":16,"value":1454,"toc":1584},[1455,1461,1465,1468,1477,1485,1488,1492,1495,1498,1501,1504,1508,1511,1515,1518,1521,1524,1534,1537,1540,1544,1547,1550,1553,1556,1560,1563,1566,1569,1576,1579,1581],[19,1456,1457,1458],{},"This article is meant for CEOs, CTOs, and Managers. The goal is to give a better understanding of what CloudFormation is and why it should be used for every AWS project, by answering a common question clients have: ",[23,1459,1460],{},"What are the the most important things our team should be aware of when working on a CF file?",[60,1462,1464],{"id":1463},"pick-a-framework","Pick a framework",[19,1466,1467],{},"Since a CloudFormation is nothing more then a JSON file, one might think that keeping everything in one big file is the right way to go. But it is not. After you have added multiple services, even as little as 3, then you will have to scroll across the file like crazy trying to edit it, and each new service exponentially extends the complexity of managing the file.",[19,1469,1470,1471,1476],{},"A good approach that I found helpful is to keep the JSON familiarity while braking down each service in a separated JSON file. Hence, I create a framework called ",[52,1472,1475],{"href":1473,"rel":1474},"https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@0x4447\u002Fgrapes",[56],"Grapes",".",[19,1478,1479,1480,1484],{},"With the ",[52,1481,1483],{"href":1473,"rel":1482},[56],"Grapes framework",", you are free to structure the Resource folder however you want. You can nest folders and name your files in total freedom – whatever makes sense to your project.",[19,1486,1487],{},"The only imposed structure is the root folders. I have used Grapes since 2019 and I have been able to make and easily manage a massive stack, since everything remains neatly organized. Below you can find a handful of links to open source projects that I released using Grapes.",[60,1489,1491],{"id":1490},"the-bare-minimum","The bare minimum",[19,1493,1494],{},"One of the few negatives about CloudFormation is the parameters part and how it is implemented in the AWS Console.",[19,1496,1497],{},"When you deploy a stack the first time with multiple parameters and there is an error, you'll loose everything you added. This is infuriating when you have a stack with 50 parameters, because if you have to redeploy a stack until it works, and let's say it takes 10 retries and you have 50 parameters, then that's equal to you having to do 500 copy and paste actions.",[19,1499,1500],{},"The only work around at this moment is to start small. Make the smallest stack that makes sense to you, deploy it successfully, and then the parameters will be saved. Afterwards you can incrementally add a service to the others.",[19,1502,1503],{},"This approach is also good to more quickly find issues within the CF file.",[60,1505,1507],{"id":1506},"good-naming-and-description","Good naming and description",[19,1509,1510],{},"Since we are on the parameters topic, I highly recommend making sure all your parameters have simple, clear names that explain what to expect from them. You can add as many details of the description as you would like, even if the description is just a copy and paste of the input field. The purpose of this is to have a clear description with your default parameters in the description itself, which gives you a reference to the parameters. If you don't have a good description, then when you come back months after you made the stack and have to deploy it in another account, then you won't remember what parameters are needed for the stack.",[60,1512,1514],{"id":1513},"do-not-rename-the-resources","Do not rename the resources",[19,1516,1517],{},"Each resource described in the CloudFormation file must have a unique name as the JSON object Key. I recommend that you name them carefully, ideally with a simple pattern, because once the stack is deploy it can cause a lot of drama to change the name of the resource. For example:",[19,1519,1520],{},"CloudFormation deploys changes by first making the new resource, and if all goes well, it deletes the old one. For example, if you have a S3 bucket that you want to rename (renaming the bucket not the service object Key), then AWS will make the new bucket first, and when all is successful, it will delete the old one if empty.",[19,1522,1523],{},"But if you were to rename the object Key, then AWS will see the changes as: \"I have to add a completely new service, while deleting the old one.\" But since AWS first makes resources and then deletes them, in this case you would get an error because AWS would try to make a S3 Bucket that already exists. Thus resulting in a update error.",[19,1525,1526,1527,1530,1531,1533],{},"Instead, the solution is to rename the resource while also renaming the S3 Bucket by adding at the end of the name, for example, ",[183,1528,1529],{},"1",". Deploying that will be sufficient, since now the buckets have unique names. Then remove the ",[183,1532,1529],{}," from the name and update the stack again.",[19,1535,1536],{},"This trick won't work in some AWS resources and you might be stuck. Make sure your resource names are standardized, and always follow this standard, because once they are used it is going to be almost impossible to rename them.",[19,1538,1539],{},"CloudFormation is for people that like to be organized and keep everything neat. With this service, you can't just do something quick and later improve upon it; you have to have it right from the start - otherwise it is going to be difficult to work with later.",[60,1541,1543],{"id":1542},"dont-refer-to-resource-outputs-in-another-aws-cloudformation-stack","Don't refer to Resource Outputs in Another AWS CloudFormation Stack",[19,1545,1546],{},"CloudFormation allows you to create a cross-stack reference that can be used in another stack. For example, one stack creates a S3 Bucket, and then you can reference the name of the Bucket in the 2nd stack.",[19,1548,1549],{},"At first this sounds like a cool and useful idea, but in practice it is the most dangerous feature out there for CF.",[19,1551,1552],{},"The problem is similar to the previous section. Because once a stack is referenced by other stack, the referenced one can't be modified because other stacks rely on it. Meaning if you have a daisy chain of 3 stacks and you wanted to edit the first one, you'd have to sequentially delete the whole stack before it, from right to left.",[19,1554,1555],{},"This feature is one that you should never use. The ideal approach when you want to reference a resource created in one stack with another, is to just output the resource name or ID in the original stack, copy it, and paste it in the params section of the second one. This way you still can split your stack and keep everything small and compact, while staying flexible and free to make any changes. Of course you have to remember that if you rename a S3 Bucket in the original stack, you then have to update all the CF files that uses the Bucket name.",[60,1557,1559],{"id":1558},"dont-trust-a-working-cloudformation-file","Don't trust a working CloudFormation file.",[19,1561,1562],{},"If you have your work organized with 2 AWS accounts, one for production and one for development, then it doesn't necessarily mean that the CloudFormation file you created and tested in development will work in production.",[19,1564,1565],{},"When working on the CF file you add features as you go, slowly building it out. And by doing so, you build on top of pre-existing services that already exist from the previous deployment.",[19,1567,1568],{},"Deploying the stack from scratch in production will create all the services at once, and in some cases a service that relies on another one won't deploy because not everything is in place, which means that the stack will fail.",[19,1570,1571,1572,1575],{},"To mitigate this problem, I recommend that you delete the stack in development and then redeploy it from scratch. This approach will let you see what will fail, and then you will know how to set the ",[183,1573,1574],{},"DependsOn"," option to make sure the stack is deployed in the right order.",[19,1577,1578],{},"Only then can you be confident that deploying the stack in production will work.",[60,1580,416],{"id":125},[19,1582,1583],{},"Move slowly, iterate piece by piece, and be organized - these are things to keep in mind when working with CloudFormation.",{"title":131,"searchDepth":132,"depth":132,"links":1585},[1586,1587,1588,1589,1590,1591,1592],{"id":1463,"depth":132,"text":1464},{"id":1490,"depth":132,"text":1491},{"id":1506,"depth":132,"text":1507},{"id":1513,"depth":132,"text":1514},{"id":1542,"depth":132,"text":1543},{"id":1558,"depth":132,"text":1559},{"id":125,"depth":132,"text":416},"knowledge","\u002Fpublic\u002Fimg\u002Farticles\u002Fcloudformation-tips-and-tricks\u002Fcover.jpg","The goal is to give a better understanding of what CloudFormation is and why it should be used for every AWS project.","\u002Fpublic\u002Fimg\u002Farticles\u002Fcloudformation-tips-and-tricks\u002Ffeatured.jpg",{},"\u002Farticles\u002Fknowledge\u002F2020-07-17-cloudformation-tips-and-tricks","\u002Fpublic\u002Fimg\u002Farticles\u002Fcloudformation-tips-and-tricks\u002Fpreview.jpg",{"title":1452,"description":1595},"articles\u002Fknowledge\u002F2020-07-17-cloudformation-tips-and-tricks","lgev99u6Q5qfUW8ZJT9y8ExnfkveqYTn3Q383ld4yGM",{"id":1604,"title":1605,"author":14,"body":1606,"category":1593,"cover":1679,"date":1680,"description":1595,"extension":142,"featured":1681,"isFeatured":434,"meta":1682,"navigation":8,"path":1683,"preview":1684,"seo":1685,"stem":1686,"__hash__":1687},"articles\u002Farticles\u002Fknowledge\u002F2020-04-08-the-importance-of-cloudformation-files.md","The Importance of CloudFormation Files",{"type":16,"value":1607,"toc":1673},[1608,1613,1617,1620,1623,1626,1630,1633,1636,1652,1656,1659,1662,1665,1668,1670],[19,1609,1457,1610],{},[23,1611,1612],{},"Why spend time making a CloudFormation file?",[60,1614,1616],{"id":1615},"what-is-cloudformation","What is CloudFormation",[19,1618,1619],{},"A CloudFormation file is nothing more than a text file with the text formatted as JSON.",[19,1621,1622],{},"Within this JSON file, you are able to describe what type of resources you'd like AWS to create for you, as if you clicked them through the AWS Console.",[19,1624,1625],{},"Simple, right?",[60,1627,1629],{"id":1628},"the-benefits-of-using-cloudformation","The Benefits of Using CloudFormation",[19,1631,1632],{},"Because everything is laid out as JSON format, when you upload this file, you get exactly what you've described.",[19,1634,1635],{},"This way...",[238,1637,1638,1641,1644],{},[241,1639,1640],{},"it's much easier to update your configuration or service, because you just have to edit one value. You will also know the exact location, instead of losing yourself to an ever changing web UI and you end up wasting time looking for an option that's buried somewhere.",[241,1642,1643],{},"if the file is committed to GitHub, you can track and monitor the changes to the stack over time.",[241,1645,1646,1647,1651],{},"you can deploy the exact stack described in the JSON file to any AWS account. This is ideal when you have multiple environments, such as one AWS account for development and another for production (learn about ",[52,1648,1650],{"href":1649},"\u002Farticles\u002Fknowledge\u002Fthe-benefits-of-aws-organizations.html","AWS Organizations"," here). What you test in one account, you can  then deploy in production within a few minutes and keep track of all your changes.",[60,1653,1655],{"id":1654},"more-flexibility","More Flexibility",[19,1657,1658],{},"Another benefit that's hard to see in the beginning, is that as you become more comfortable with CloudFormation files you will start to like this way of working in the cloud.",[19,1660,1661],{},"After you've done one file (which takes some time at first), the next time you make a new project based on a similar cloud structure it will take just hours or a few days, instead of weeks or months.",[19,1663,1664],{},"You can just copy and past what you need from your completed file. The best thing is that once a file like this is created and tested, you know it works – now and in the future. And, even if AWS makes some changes, modifications to the file will be minor and easy to do.",[19,1666,1667],{},"It takes time and money to get the hang of it, but once you do you'll have a nice foundation to build upon and extend.",[60,1669,416],{"id":125},[19,1671,1672],{},"I hope that you now understand the benefits of using CloudFormation files and that this AWS Service will help you streamline your DevOps work flow.",{"title":131,"searchDepth":132,"depth":132,"links":1674},[1675,1676,1677,1678],{"id":1615,"depth":132,"text":1616},{"id":1628,"depth":132,"text":1629},{"id":1654,"depth":132,"text":1655},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-importance-of-cloudformation-files\u002Fcover.jpg","2020-04-08","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-importance-of-cloudformation-files\u002Ffeatured.jpg",{},"\u002Farticles\u002Fknowledge\u002F2020-04-08-the-importance-of-cloudformation-files","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-importance-of-cloudformation-files\u002Fpreview.jpg",{"title":1605,"description":1595},"articles\u002Fknowledge\u002F2020-04-08-the-importance-of-cloudformation-files","yIHmVUO1nALRiZfOvyJw2avVteQSICBiuBhZbj5NLcs",{"id":1689,"title":1690,"author":14,"body":1691,"category":1593,"cover":1789,"date":1790,"description":1791,"extension":142,"featured":1792,"isFeatured":434,"meta":1793,"navigation":8,"path":1794,"preview":1795,"seo":1796,"stem":1797,"__hash__":1798},"articles\u002Farticles\u002Fknowledge\u002F2020-03-28-should-i-go-all-in-with-serverless.md","Should I go all in with Serverless?",{"type":16,"value":1692,"toc":1780},[1693,1699,1703,1706,1709,1713,1716,1720,1728,1731,1739,1742,1745,1749,1752,1756,1759,1763,1766,1775,1777],[19,1694,1695,1696],{},"This article is meant for CEOs, CTOs, and managers who want to make a more educated decision about building products in the cloud. We will answer a common question clients have: ",[23,1697,1698],{},"Should I go serverless or build a project with regular servers?",[60,1700,1702],{"id":1701},"first","First",[19,1704,1705],{},"There isn't a short answer to this question, because it all depends on the stage of your project. I always recommend that new clients start out serverless when in the initial phases of exploring an idea that may or may not work, meaning when you are not certain how much traffic the site is going to get. Then, once you begin to get some decent traffic, you are able to better predict how much load the solution will get, and thus transition some of the most expensive parts of the project to a regular server and lower the costs of running the solution. After your business is established, the final transition stage would likely be 90 percent use of servers with some serverless remaining for anything that isn't triggered often or when only used to manage DevOps daily tasks.",[19,1707,1708],{},"So the question becomes: Why this life cycle?",[60,1710,1712],{"id":1711},"start-with-serverless","Start with serverless",[19,1714,1715],{},"One advantage of starting serverless is that is is not expensive at the outset of the project. It will cost you virtually nothing to run during development, regardless of how many resources you use. In the beginning phase as you are acquiring customers, all you pay for is traffic generated. If you have to pause the project for any reason, then you don't have to delete anything or worry about starting over, because you can preserve the entire set up and just pay for storage. Then, when all is back to normal, you can resume the project right where you left it.",[60,1717,1719],{"id":1718},"move-to-managed-servers-to-lower-costs","Move to managed servers to lower costs",[19,1721,1722,1723,1727],{},"Serverless may be convenient in the beginning when you're testing the waters, but it's definitely not cheap in the long run. Once you see your idea is catching on and you're getting more and more traffic to your product, you should review your AWS bill on a monthly basis to identify the most expensive serverless resources. Use your list of the most expensive resources to begin converting to a custom server setup using EC2 Instances, but be sure that you understand the ",[52,1724,1726],{"href":1725},"\u002Farticles\u002Fhow-to\u002Fhow-to-lower-ec2-costs.html","benefits of reserved instances"," in order to know that you are paying as little as possible for them.",[19,1729,1730],{},"A common misconception about using physical servers is that they requires dedicated people to monitor and react when something is not working as expected. Thankfully that way of thinking is old fashioned and doesn't take into account the current programmatic nature of AWS. Nowadays you can manage the whole infrastructure with code and automate many of the moving parts.",[19,1732,1733,1734,1738],{},"By using ",[52,1735,1737],{"href":1736},"\u002Farticles\u002Fknowledge\u002Fthe-importance-of-cloudformation-files.html","CloudFormation"," and all the EC2 tools provided, you can create a server environment that acts like a serverless one. Operation is seamless and developers just have to push new code to a repository to make the stack automatically pull the changes and replace the old code, even without anyone having physical access to the servers.",[19,1740,1741],{},"Another important factor: a setup like this implies no need for physical access to the deployed servers, no SSH Keys, or Port 22 open, etc. Once set up correctly, everything should run smoothly. If AWS is able to operate your servers, then you should be able to as well, because serverless ends up being no more involved than a regular server set up in a organized and automated way. Serverless could be considered just a concept.",[19,1743,1744],{},"If you have migrated to EC2 Instances, then you can monitor the AWS bills and continue to transfer more and more resources until you find the price balance that's right for you.",[60,1746,1748],{"id":1747},"volume-discount","Volume discount",[19,1750,1751],{},"With the right price balance in hand and your whole setup working correctly, you can take additional steps to lower your costs by contacting the AWS Pricing Team to see if they can give you a volume discount for some of the services you're using.",[60,1753,1755],{"id":1754},"think-about-going-outside-aws","Think about going outside AWS",[19,1757,1758],{},"You can also look to see what's outside AWS. There are cloud providers out there that specialize in providing the cheapest price for specific AWS services. For example, Wasabi competes directly against S3. Their prices are truly much cheaper compared with what AWS has to offer, but you also have to consider that AWS has more regions available, more storage options for S3 to lower your prices, and more features - including triggers. This means that it all depends on your needs. But if what Wasabi offers is perfect for your solution, why not consider them?",[60,1760,1762],{"id":1761},"go-custom-data-center","Go custom data center",[19,1764,1765],{},"And now, the last major step: a lot of online articles like to imply that AWS is too expensive and that having your our own data center is the cheapest way for you to serve a product to your clients. But this statement will only be true for a select few companies that have either outgrown the AWS volume discount, operate at a scale close to that of AWS, or only interact with a local market.",[19,1767,1768,1769,1774],{},"A good example would be ",[52,1770,1773],{"href":1771,"rel":1772},"https:\u002F\u002Fwww.businessinsider.com\u002Fbank-of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10",[56],"Bank of America",", which transitioned to their own data center to lower costs. For a company of this size that deals only with the local market, this makes perfect sense and is the right move.",[60,1776,416],{"id":125},[19,1778,1779],{},"I hope this explanation will help you organize your thoughts around the serverless approach and you'll know exactly how to discuss this with your team or contractors.",{"title":131,"searchDepth":132,"depth":132,"links":1781},[1782,1783,1784,1785,1786,1787,1788],{"id":1701,"depth":132,"text":1702},{"id":1711,"depth":132,"text":1712},{"id":1718,"depth":132,"text":1719},{"id":1747,"depth":132,"text":1748},{"id":1754,"depth":132,"text":1755},{"id":1761,"depth":132,"text":1762},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fshould-i-go-all-in-with-serverless\u002Fcover.jpg","2020-03-28","The goal is to better understand Serverless technology and how it affects your costs as your project grows.","\u002Fpublic\u002Fimg\u002Farticles\u002Fshould-i-go-all-in-with-serverless\u002Ffeatured.jpg",{},"\u002Farticles\u002Fknowledge\u002F2020-03-28-should-i-go-all-in-with-serverless","\u002Fpublic\u002Fimg\u002Farticles\u002Fshould-i-go-all-in-with-serverless\u002Fpreview.jpg",{"title":1690,"description":1791},"articles\u002Fknowledge\u002F2020-03-28-should-i-go-all-in-with-serverless","Vo830ojtEdqHVcdTyvixIxi54cbitnOkxQb7R811bpE",{"id":1800,"title":1801,"author":14,"body":1802,"category":1593,"cover":2035,"date":2036,"description":2037,"extension":142,"featured":2038,"isFeatured":434,"meta":2039,"navigation":8,"path":2040,"preview":2041,"seo":2042,"stem":2043,"__hash__":2044},"articles\u002Farticles\u002Fknowledge\u002F2020-03-15-the-benefits-of-aws-organizations.md","The benefits of AWS organizations",{"type":16,"value":1803,"toc":2025},[1804,1810,1814,1817,1834,1837,1841,1858,1860,1863,1880,1884,1887,1910,1918,1921,1925,1928,1931,1934,1938,1952,1956,1959,2020,2022],[19,1805,1806,1807],{},"This article is meant for CEOs, CTOs, and managers who want to understand the benefits of managing AWS infrastructure using Organizations. We will answer a common question clients have: ",[23,1808,1809],{},"Why spend the time setting up AWS Organizations?",[60,1811,1813],{"id":1812},"why-i-wrote-this-article","Why I wrote this article",[19,1815,1816],{},"I have worked with countless clients on recurring issues that leave them feeling trapped and struggling how to:",[238,1818,1819,1822,1825,1828,1831],{},[241,1820,1821],{},"building new features",[241,1823,1824],{},"understanding where they are",[241,1826,1827],{},"figuring out whether their infrastructure is secure",[241,1829,1830],{},"running a stable product",[241,1832,1833],{},"Understand their costs",[19,1835,1836],{},"I am confident that this article will help you understand how important is to use AWS at the outset so that you start off organized and thus avoid the extra time and costs involved in having to later reorganize.",[60,1838,1840],{"id":1839},"key-features-of-aws-organizations","Key features of AWS Organizations",[238,1842,1843,1846,1849,1852,1855],{},[241,1844,1845],{},"You have one bill to pay for all of your AWS accounts within Organizations.",[241,1847,1848],{},"You get a detailed breakdown of your bill per AWS account, which helps showcase where your money goes.",[241,1850,1851],{},"You can control access to AWS resources for each individual AWS account from the main Organizations account.",[241,1853,1854],{},"You can apply policies from the Organizations account down to all the sub-accounts.",[241,1856,1857],{},"You can organize the sub-accounts in Organization Unit (OU) to help you further organize the company in AWS and improve security.",[60,1859,491],{"id":490},[19,1861,1862],{},"It's much easier to:",[238,1864,1865,1868,1871,1874,1877],{},[241,1866,1867],{},"Do audits",[241,1869,1870],{},"Set the right policies and access resources",[241,1872,1873],{},"Create test environments",[241,1875,1876],{},"Understand what each AWS account has for newcomers",[241,1878,1879],{},"Lower costs by exposing hidden costs",[60,1881,1883],{"id":1882},"the-bare-minimum-setup","The bare-minimum setup",[19,1885,1886],{},"I always recommend this setup to my clients:",[238,1888,1889,1896,1903],{},[241,1890,1891,1892,1895],{},"In ",[244,1893,1894],{},"Development"," you test and work on the project. Here you have the most freedom to use different resources.",[241,1897,1898,1899,1902],{},"The ",[244,1900,1901],{},"Demo"," account is where you can showcase the latest features that are ready to go into production. This is your last chance to see if you like what's been done. It can also be used to perform demos for clients.",[241,1904,1905,1906,1909],{},"On the other hand, ",[244,1907,1908],{},"Production"," is for your customers with live data.",[19,1911,1898,1912,1914,1915,1917],{},[244,1913,1901],{}," and ",[244,1916,1908],{}," environments are locked down to the point that no one but the main admin has access to them.",[19,1919,1920],{},"All accounts have 100 percent automatic deployment and do not have access to the servers over SSH or other means – no exceptions.",[60,1922,1924],{"id":1923},"a-word-of-caution","A word of caution",[19,1926,1927],{},"Having said all this, here's an example of what will happen if you start the wrong way, based on interacting with clients over the years.",[19,1929,1930],{},"Depending on project size, creating a CloudFormation from scratch takes an average of one month, and then it takes two more weeks to create all the AWS accounts to deploy the stack using the created CloudFormation files. This is the stage where all the final testing and final tuning happens.",[19,1932,1933],{},"Lastly, you'll start running your code in the new environment with a new work flow. A lot of the project's bugs will surface at this stage, because up to this point no one could log into the servers to perform ad hoc changes (also known as massaging the code). Now all the work a developer told himself he'd copy later in the repository shows up at this stage. How long will it take to fix everything? It depends on the project. From my experience, it takes a team of developers three months to normalize everything and adapt to the more organized environment.",[60,1935,1937],{"id":1936},"what-else-should-i-know","What else should I know?",[238,1939,1940,1943,1946,1949],{},[241,1941,1942],{},"You're limited to five nested OU, so design your infrastructure accordingly.",[241,1944,1945],{},"You can use CloudFormation to describe your organization.",[241,1947,1948],{},"You can programmatically request a limit increase.",[241,1950,1951],{},"Converting the main account to an Organization won't make any changes to what you already have in the account.",[60,1953,1955],{"id":1954},"an-example-of-a-more-complex-organization","An example of a more complex Organization",[19,1957,1958],{},"This is an example for once you get your project off the ground. You could extend your Organization this way later on:",[238,1960,1961,1975,1986,2000,2017],{},[241,1962,1963,1964],{},"Application (OU):\n",[238,1965,1966,1969,1972],{},[241,1967,1968],{},"Development - (AWS Account)",[241,1970,1971],{},"Demo - (AWS Account)",[241,1973,1974],{},"Production - (AWS Account)",[241,1976,1977,1978],{},"Information security (OU):\n",[238,1979,1980,1983],{},[241,1981,1982],{},"Log Collection - (AWS Account)",[241,1984,1985],{},"Auditing - (AWS Account)",[241,1987,1988,1989],{},"Infrastructure (OU):\n",[238,1990,1991,1994,1997],{},[241,1992,1993],{},"DNS - (AWS Account)",[241,1995,1996],{},"Build Tools - (AWS Account)",[241,1998,1999],{},"Repos - (AWS Account)",[241,2001,2002,2003],{},"Office (OU):\n",[238,2004,2005,2008,2011,2014],{},[241,2006,2007],{},"Work Mail - (AWS Account)",[241,2009,2010],{},"VPN - (AWS Account)",[241,2012,2013],{},"NAS - (AWS Account)",[241,2015,2016],{},"AppStream - (AWS Account)",[241,2018,2019],{},"Etc.",[60,2021,416],{"id":125},[19,2023,2024],{},"I hope this explanation helps you organize your thoughts around AWS Organizations so that you can know exactly what to ask for when talking to contractors or your team.",{"title":131,"searchDepth":132,"depth":132,"links":2026},[2027,2028,2029,2030,2031,2032,2033,2034],{"id":1812,"depth":132,"text":1813},{"id":1839,"depth":132,"text":1840},{"id":490,"depth":132,"text":491},{"id":1882,"depth":132,"text":1883},{"id":1923,"depth":132,"text":1924},{"id":1936,"depth":132,"text":1937},{"id":1954,"depth":132,"text":1955},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-benefits-of-aws-organizations\u002Fcover.jpg","2020-03-15","The benefits of managing your AWS infrastructure using Organizations.","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-benefits-of-aws-organizations\u002Ffeatured.jpg",{},"\u002Farticles\u002Fknowledge\u002F2020-03-15-the-benefits-of-aws-organizations","\u002Fpublic\u002Fimg\u002Farticles\u002Fthe-benefits-of-aws-organizations\u002Fpreview.jpg",{"title":1801,"description":2037},"articles\u002Fknowledge\u002F2020-03-15-the-benefits-of-aws-organizations","Xyk0CwxQXhaifjerGV2IolmBjepS0fZ2XHHT78viI1I",{"id":2046,"title":2047,"author":14,"body":2048,"category":138,"cover":2170,"date":2036,"description":2171,"extension":142,"featured":2172,"isFeatured":434,"meta":2173,"navigation":8,"path":2174,"preview":2175,"seo":2176,"stem":2177,"__hash__":2178},"articles\u002Farticles\u002Fhow_to\u002F2020-03-15-how-to-lower-ec2-costs.md","How to lower EC2 Costs",{"type":16,"value":2049,"toc":2165},[2050,2056,2060,2063,2065,2070,2073,2077,2080,2118,2121,2124,2128,2131,2135,2138,2142,2145,2148,2151,2154,2157,2160,2162],[19,2051,2052,2053],{},"This article is meant for CEOs, CTOs, and Managers. The goal is lowering your AWS costs by 40 or even 75 percent within a few minutes (depending on the instance type and plan), by answering a common question clients have: ",[23,2054,2055],{},"What options do I have at my disposal to lower the AWS bill?",[60,2057,2059],{"id":2058},"what-are-reserved-instances","What are Reserved Instances?",[19,2061,2062],{},"If you pay for an instance upfront for a period of either one or three years, then your price for an instance can be 40 to 75 percent lower. The beauty of this is that if you buy a Reserved Instance, you don't have to make changes to your instances - no restart or reconfiguration is necessary. This is because a Reserved Instance is a billing concept. If you have two instances and you buy one Reserved Instance, AWS will automatically apply the discount for the instance that matches the type you bought. The entire process is seamless for you.",[60,2064,1937],{"id":1936},[2066,2067,2069],"h4",{"id":2068},"standard-vs-convertible","Standard vs. Convertible",[19,2071,2072],{},"There are two types of Reserved Instances: Standard and Convertible. Standard is suitable for a situation when you have an organized environment and you know what you need. Convertible (a bit more  expensive) is for situations with a more dynamic environment that requires constant changes to your operating systems and instance families as well as types.",[2066,2074,2076],{"id":2075},"normalization-factor","Normalization Factor",[19,2078,2079],{},"AWS gives you quite a bit of flexibility with the Normalization Factor approach. Each instance type has the following factors:",[238,2081,2082,2085,2088,2091,2094,2097,2100,2103,2106,2109,2112,2115],{},[241,2083,2084],{},"nano: 0.25",[241,2086,2087],{},"micro: 0.5",[241,2089,2090],{},"small: 1",[241,2092,2093],{},"medium: 2",[241,2095,2096],{},"large: 4",[241,2098,2099],{},"xlarge: 8",[241,2101,2102],{},"2xlarge: 16",[241,2104,2105],{},"4xlarge: 32",[241,2107,2108],{},"8xlarge: 64",[241,2110,2111],{},"10xlarge: 80",[241,2113,2114],{},"16xlarge: 128",[241,2116,2117],{},"32xlarge: 256",[19,2119,2120],{},"This means that if you buy one large type, you can actually have four small instance types, or two small ones and four nano types. You can mix this as long as the total factor number matches the size of what you bought.",[19,2122,2123],{},"AWS also allows you to scale above what you have. For example, let's say you bought a large type for the year and you scaled up to the 2xlarge type for one week. AWS will charge you the On-Demand price, but only for the scaling factor that goes beyond what you paid for. They do the calculation automatically so you don't have to worry about it.",[2066,2125,2127],{"id":2126},"be-organized","Be organized",[19,2129,2130],{},"The Scaling Factor is flexible, but best practice is to have a homogeneous instance type. This makes it much easier to manage and keeps costs to a minimum.",[2066,2132,2134],{"id":2133},"you-can-resell-what-you-dont-use","You can resell what you don't use",[19,2136,2137],{},"If you buy a one-year or three-year lease, and after some time you find that you don't need it anymore, or you've upgraded to a new type because of traffic to your site, then you can buy the new type and put the old lease on the market for bidding. While you won't be able to get 100 percent of the money back from the remainder of your old type, you'll still get a decent amount.",[2066,2139,2141],{"id":2140},"the-roll-over-strategy","The roll-over strategy",[19,2143,2144],{},"Even with the flexibility AWS gives you, you may still be left with some unused instance types. Before you resell them, you should consider the following strategy: roll your instance types to the next-lower traffic site. For example, let's assume that you have a site with two parts, the front end and the back end. The back is the API, and the front delivers the site build using Angular, with some server rendering for good SEO.",[19,2146,2147],{},"Both projects run using a load balancer, and each LB has a minimum of two instances of type t3.small. You buy one type, a t3.large server, for one year and pay 100 percent upfront. You then use the Normalization Factor to fragment the one large into four small types.",[19,2149,2150],{},"Six months later you get so much traffic that you have to upgrade the back end to a larger type. In this case, you could change the instance type by buying a one-year lease for a large type and split that into two mediums.",[19,2152,2153],{},"Now you have two units to spare. You could add two more small (or one medium) instance types to your front end to handle the additional traffic.",[19,2155,2156],{},"The reasoning here is that more traffic will increase the load on both projects, but the front end will have the smallest load, and that can be handled by four small server types.",[19,2158,2159],{},"When the lease for the old types expires, you can switch the front end to two bigger server types or extend the lease for what you already have. AWS's flexibility means you have options to role-over instance types and save money.",[60,2161,416],{"id":125},[19,2163,2164],{},"Stop using On-Demand servers for long-term work. Use them only for quick tests, maintenance, or temporary auto scaling to handle spikes in traffic during the day. For continuous work, use Reserved Instances, and spend your money elsewhere.",{"title":131,"searchDepth":132,"depth":132,"links":2166},[2167,2168,2169],{"id":2058,"depth":132,"text":2059},{"id":1936,"depth":132,"text":1937},{"id":125,"depth":132,"text":416},"\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-ec2-costs\u002Fcover.jpg","Lower your AWS costs by 40 or even 75 percent within a few minutes (depending on the instance type and plan).","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-ec2-costs\u002Ffeatured.jpg",{},"\u002Farticles\u002Fhow_to\u002F2020-03-15-how-to-lower-ec2-costs","\u002Fpublic\u002Fimg\u002Farticles\u002Fhow-to-lower-ec2-costs\u002Fpreview.jpg",{"title":2047,"description":2171},"articles\u002Fhow_to\u002F2020-03-15-how-to-lower-ec2-costs","aAnsTuXM2SDP4gTx6kv3lblrRphQ_0j4cl7ZIfoHeLA",1783756517015]